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Combat  Service  Support  (CSS)  Training  System  Development 
for  the  U.S.  Army  National  Guard  (ARNG) 

Final  Report 

INTRODUCTION 


Background 

By  1992,  information  had  become  available  indicating  that  there  were  problems  in  the 
training  readiness  level  of  the  U.S.  Army  National  Guard  (ARNG).  Preparations  for  the  Gulf 
War  (Operations  Desert  Shield  and  Desert  Storm)  had  demonstrated  that  larger  (battalion  and 
above)  reserve  combat  units  would  require  considerable  intensive  training  to  become  combat 
ready.  Trends  reported  from  the  Army’s  Comhat  Training  Centers  indicated  that  casualty 
evacuation  and  synchronization  of  Combat  Service  Support  (CSS)  were  persisting  weaknesses'. 

One  fundamental  problem  with  ARNG  training  is  that  there  are  only  12  weekends  and 
one  1 5-day  period  of  active  training  each  year  —  a  total  of  39  training  days.  The  geographic 
dispersion  of  the  guardsmen  relative  to  their  training  center  further  reduces  the  effective  training 
time  in  some  cases.  Guardsmen  in  Forward  Support  Battalions  (FSBs)  are  not  given  adequate 
access  to  the  actual  equipment  used  in  performance  of  their  duties.  This  lack  of  realism  inhibits 
effective  training. 

One  of  the  initiatives  directed  to  address  these  problems  was  Congressionally-mandated 
development  of  improved  training  within  the  constraints  faced  by  ARNG  units  through  the  use  of 
computer-based  technologies.  The  ARNG  and  the  Advanced  Research  Projects  Agency  (ARP A) 
cooperated  to  sponsor  the  development  of  this  technology,  one  focus  of  which  was  on  improved 
training  for  ARNG  FSBs. 

Two  broad  areas  for  development  of  training  for  FSB  personnel  were  identified: 

1)  Training  for  individuals  in  the  three  operational  companies  of  the  FSB:  Supply, 
Maintenance,  and  Medical. 

2)  Training  for  the  FSB  staff,  which  is  responsible  for  coordination  and  synchronization 
of  efforts  to  support  all  of  the  units  under  the  control  of  the  Brigade. 

ARPA  and  the  ARNG  determined  that  a  lanes-based  training  approach  developed  by  the 
Iowa  National  guard  (lANG)  was  a  suitable  paradigm  for  the  development  of  training  for 
personnel  in  FSB  operational  companies:  Supply,  Maintenance,  and  Medical.  Consequently, 
ARPA  contracted  with  the  lANG  to  have  certain  of  their  personnel  detailed  full-time  to  this 
project  to  provide  subject  matter  expertise  for  both  the  lanes-based  and  staff  training.  The  lANG 


'  Video  Teleconference  on  Combat  Training  Center  Trends  conducted  by  the  Center  for 
Army  Lessons  Learned  on  21  Oct  1992. 


also  contributed  additional  personnel  to  this  effort,  and  supplied  office  space  and  other  support. 
The  eontractor  supplied  knowledge  of  instructional  development,  experience  in  the  development 
of  computer-based  instruction,  and  experience  in  the  development  of  computerized  systems  to 
manage  instruction. 

The  remainder  of  this  report  documents  the  development  of  instruction  in  each  of  these 
two  areas.  Diseussion  in  each  area  of  instruction  examines:  the  selection  of  topics  to  train,  the 
steps  in  developing  the  instruction,  and  the  formative  evaluation  of  the  lessons.  A  separate 
section  is  devoted  to  the  system  developed  to  manage  the  instruction.  A  concluding  section  on 
lessons  learned  addresses  improvements  and  efficiencies  that  could  be  realized  in  future  efforts 
of  this  nature.  Supplementary  appendices  provide  background  material  on  the  nature  of 
computer-based  instruction  and  information  about  the  staffing  and  management  of  the  project. 

DESIGN  AND  DEVELOPMENT  OF  TRAINING  FOR  FSB  OPERATIONAL  COMPANIES 
Lane  Lesson  Front  End  Analysis  and  Design 

The  development  of  specifications  for  the  lane  training  lessons  was  accomplished  by  the 
ARPA  and  lANG  subject  matter  experts  (SMEs)  at  the  beginning  of  the  project,  during  the 
Spring  and  Summer  of  1993.  “Lane  training”  originated  in  infantry  units  in  the  Army.  Typical 
infantry  training  lanes  require  squads,  platoons,  and  companies  to  negotiate  over  a  designated 
piece  of  terrain  in  order  to  accomplish  a  specified  mission.  The  units  start  in  an  assembly  area, 
eross  a  line  of  departure,  and  navigate  over  a  predetermined  route  to  an  objective,  thus  the  term 
“lane  training”.  Use  of  the  term  by  other,  non-maneuver,  types  of  units  has  resulted  in  some 
confusion  over  the  nature  of  the  training  to  which  it  refers. 

Today  the  lane  training  strategy  is  used  in  all  types  of  Army  units.  When  applied  in  other 
units,  the  term  "lane  training"  is  somewhat  of  a  misnomer,  because  in  many  cases  the  "lane"  is 
one  static  location.  The  broader  definition  of  lane  training  is:  a  technique  for  identifying, 
specifying,  and  training  company  and  subordinate  platoons  in  a  series  of  related  leader  and 
individual  tasks  that  are  critical  to  the  execution  of  mission  essential  tasks.  Lanes  were 
developed  for  this  project  using  a  top  down  analysis  of  missions,  critical  collective  sub-tasks,  as 
well  as  supporting  leader  and  individual  tasks.  This  pyramidal  approach  allowed  the  SMEs  to 
filter  critical  tasks  from  the  myriad  of  knowledge,  skills,  and  abilities  which  seemingly  carry  the 
same  level  of  importance. 

Project  SMEs  had  experience  with  training  lanes  developed  for  maintenance  persoimel  by 
the  lANG.  These  lanes  take  advantage  of  the  fact  that  Camp  Dodge  has  the  largest  CONUS 
maintenance  training  facility.  Actual  equipment  to  be  maintained  is  brought  onto  the  shop  floor 
and  all  of  the  necessary  tools  and  manuals  are  assembled  into  a  lane  package.  Trainees  then  are 
coached  through  the  performance  of  the  required  maintenance:  they  receive  more  coaching  in  the 
‘crawl’  stage  of  training,  and  progressively  less  as  they  proceed  through  ‘walk’  and  ‘run’  stages. 
This  approach  is  very  resource  intensive  and  is  difficult  to  duplicate  in  ARNG  armories. 
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Computer-based  instruction  (CBI)  was  to  be  used,  where  appropriate,  to  provide  effective 
instruction  without  the  requirement  of  so  many  resources. 

The  first  objective  for  the  project  SMEs  was  to  perform  a  similar  top-down  analysis  of 
the  collective  tasks  in  each  company  that  would  lead  to  the  identification  of  a  training  lane.  Then 
the  tasks  vdthin  the  lane  would  be  further  evaluated  to  determine  which  of  them  were  most 
suitable  to  training  using  CBI. 

The  project  SMEs  began  by  identifying  the  mission  essential  tasks  to  be  performed  by 
each  of  the  operational  companies  of  an  FSB.  From  these  they  selected  one  task  for  each 
company  for  further  development.  In  addition,  they  identified  Defend  Company  Sector  as  a 
mission  essential  task  that  applied  to  all  three  companies.  The  unit  Mission  Training  Plans 
(MTPs)  were  consulted  to  identify  the  collective  tasks  that  make  up  the  mission  essential  tasks 
chosen  for  each  company.  The  most  critical  collective  tasks  were  selected  and  verified  by  the 
National  Training  Center.  To  identify  the  topics  for  those  lessons,  the  leader  and  individual  tasks 
that  must  be  accomplished  to  enable  the  unit  to  perform  the  most  critical  collective  tasks  were 
identified  next  (using  the  MTPs  and  Soldier  Training  Plans). 

A  quantitative  approach  was  used  to  select  tasks  for  CBI  development.  ARPA  staff 
formulated  decision  criteria  to  derive  a  numerical  priority  value  for  each  of  the  individual  and 
leader  tasks  that  were  critical  to  each  lane.  Table  1  identifies  the  criteria  or  conditions  that  were 
used  in  the  decision  process.  The  criteria  are  a  combination  of  performance,  training,  and 
practical  conditions  within  which  CBI  could  offer  significant  advantage.  By  application  of  these 
criteria,  the  SMEs  identified  tasks  within  each  company's  selected  collective  task  that  appeared 
appropriate  for  CBI  development.  The  most  important  criteria  in  making  this  selection  related  to 
the  difficulty  of  performing  the  training  in  typical  armory  settings.  These  tasks  were  structured 
into  topics  within  each  lesson. 

In  order  to  provide  approximate  estimates  of  the  amount  of  CBI  to  be  developed,  the 
SMEs  estimated  the  student  completion  time  for  each  lesson.  These  estimates  were  based  on  the 
SMEs’  prior  experience  with  traditional  training  methods,  as  none  of  them  had  previous 
experience  in  making  such  estimates  for  individualized,  self-paced  lessons. 

The  SMEs  also  prepared  descriptions  of  the  training  objectives  for  each  task  or  cluster  of 
tasks.  The  objectives  specified  the  actions  to  be  performed,  the  conditions  under  which  the 
performance  would  take  place  in  the  field  (including  tools,  equipment,  and  other  essential 
materials),  and  the  standards  of  acceptable  performance.  The  SMEs  obtained  and  assembled 
"lane  books"  that  contained  various  doctrinal  and  relevant  training  and  reference  materials. 
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Table  1 


Criteria  for  selecting  tasks  for  implementation  via  CBI. 

SPECIFIC  CRITERIA: 

•  COMPLEXITY  OF  TASK  (difficult  of  traditional  training  due  to  the  skill  required). 

•  LOW  DENSITY  CRITICAL  TASKS  (tasks  that  are  critical  to  the  mission  but  involve 
only  a  few  individuals). 

•  EQUIPMENT  SHORTAGE  (lack  of  equipment  on  which  to  train) 

•  SUPPORT  EQUIPMENT  (lack  of  special  tools  or  test  equipment  to  conduct  training). 

•  LACK  OF  TRAINERS  (sufficient  numbers  of  qualified  trainers  are  not  available). 

•  TIME  INTENSIVE  (an  extensive  amount  of  time  is  required  to  set  up  and  conduct 
training). 

•  SAFETY  (trainees  can  leam  potentially  danagerous  tasks  without  risk). 

GENERAL  CRITERIA: 

•  REDUCED  LEARNING  TIME  (interactive  learning  is  up  to  50%  more  efficient  than 
traditional  training  techniques). 

•  REDUCED  COST  (compared  to  alternatives). 

•  INSTRUCTIONAL  CONSISTENCY  (compared  to  alternatives;  training  provided  by  each 
first  line  supervisor  may  be  inconsistent  in  quality  and  accuracy). 

•  PRIVACY  (peer  pressure  can  adversely  affect  learning). 

•  MASTERY  LEARNING  (structured  practice  and  self  pacing  provide  a  path  for  logical 
learning). 

•  INCREASED  RETENTION  (interaction  between  the  trainee  and  CBI  provides  increased 
retention  over  time). 

•  INCREASED  MOTIVATION  (interactive  learning  has  been  shown  to  be  highly 
motivational  for  learners). 

•  INCREASED  ACCESS  (quality  training  can  be  available  to  the  trainee  at  a  time  and  place 
convenient  for  the  trainee). 


Appendix  A  shows  the  original  conceptualization  of  the  lessons  for  each  of  the  training  lanes. 
During  the  production  of  the  lessons  some  changes  were  made:  Lesson  4  in  the  Supply  lane  was 
too  long  and  was  split  into  lessons  4  and  5;  three  of  the  Defend  Company  Sector  lessons 
combined  tasks  for  leaders  and  other  personnel  dangerous  felt  that  these  should  be  separated. 
The  final  lessons  and  their  titles  are  listed  in  Table  2^. 


^  A  ‘collective’  lesson  was  planned  to  supplement  each  of  the  medical  company  and  the 
Defend  Sector  lanes.  Ultimately,  this  requirement  was  dropped,  for  reasons  discussed  later. 
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Table! 


List  of  Training  Lane  CBI  courses  for  operational  companies  in  an  FSB. 


Supply  Lane  A-1  Supervise  Receipt/Storage  of  POL 

A-2  Inspection  of  POL  Products 
A-3  Direct  POL  Environment  and  Security  Controls 
A-4  Tanker  Operations  and  Safety 
A-5  Tank  Trailer  Operations 

Maintenance  Lane  B-1  Inspect  and  Troubleshoot  Tracked  Vehicles 
B-2  Repair  Diesel  Power  Plant/Pack 
B-3  Test  and  Troubleshoot  Radio  Sets 
B-4  Repair  Traversing  Systems 
B-5  Repair/Replace  TOW  System  on  BFV 
B-6  Organize  and  Dispatch  an  MST 
B-7  Battle  Damage  Assessment  and  Repair  (BDAR) 

Medical  Lane  C-1  Control  Bleeding 

C-2  Survey  and  Triage 
C-3  Plan  for  Evacuation  Support 
C-4  Airway  Management 
C-5  General  Casualty  Management 
C-6  Treatment  of  Wounds 


Defend  Sector  Lane  D-1  Analyzing  Terrain 
D-2  Plan  Sector  Defense 
D-3  Prepare  Support  Plan 
D-4  Prepare  for  Engagement 
D-5  Company  Prepares  for  Engagement 
D-6  Co  CDR  Organizes  Hasty  Displacement/Disengagement 
D-7  Hasty  Displacement/Disengagement  Under  Fire 


Lesson  Development  and  Authoring 

To  be  successful,  the  project  had  to  blend  the  subject  matter  expertise  of  the  ARPA  and 
lANG  staff  with  the  comse  development  and  CBI  authoring  expertise  of  the  contractor  staff. 
Since  the  latter  had,  essentially,  no  prior  experience  with  the  Army,  the  ARPA  and  lANG  SMEs 
had  to  supply  all  of  the  subject  matter  information:  Doctrinally  correct  statements  of  tasks, 
conditions  and  standards,  and  narratives  linking  them  into  appropriate  sequences.  They  also  had 
to  define  the  objectives  of  instruction,  make  the  desired  terminal  performance  clear  (for  both  the 
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CBI  authors  and,  ultimately,  the  students),  and  provide  assistance  with  developing  criterion  tests 
to  be  used  to  assess  entry  level  skills  and  monitor  student  progress. 

The  contractor  developed  a  17-step  sequence  for  developing  each  of  the  lessons,  shown  in 
Table  3.  This  sequence  assisted  the  managers  to  allocate  responsibilities  among  the  SMEs,  the 
course  developers,  and  the  CBI  authors,  and  facilitated  tracking  course  development  progress. 
The  general  lesson  development  process  outlined  in  Table  3  was  followed  throughout  the  project 
with  certain  modifications,  described  below. 

Table  3 

Seventeen-step  developmental  sequence  for  CBI  lessons. 


1.  SME;  Does  Initial  Analysis  and  Objectives 

2.  SME:  Prepares  Content  Narratives 

3.  Developer  and  SME:  Develop  Criterion  Test 

4.  Developer  and  SME:  Identify  Text  and  AV  Specs  and  Sources 

5.  QA  Criterion  Test  and  Text^AV  Specs 

6.  SMEs  working  on  other  lessons:  Review  for  Technical  Accuracy  and  Completeness 

7.  Developer:  Prepare  Lesson  Design 

8.  Developer:  Completes  Interactive  Storyboards 

9.  SME:  Reviews  for  Technical  Accuracy 

10.  SME,  Developer,  CBI  Author:  Story  board  Conference 

1 1 .  CBI  Author:  Converts  Storyboards  to  CBI 

12.  SMEs:  Alpha  Test  Lessons,  QA  CBI  Draft,  Specify  Revisions 

13.  CBI  Author:  Revise  CBI  Test  Version 

14.  Developer/SME:  Beta  Test:  Conduct  Individual  Trials,  2/3  Soldiers 

15.  SME  and  CBI  Author:  Specify  Revisions 

16.  CBI  Author:  Revise,  Retest 

17.  Deliver  Final  Version  to  PRC,  Monterey 


Some  SMEs  were  not  able,  for  a  variety  of  reasons,  to  provide  a  written  content  narrative. 
They  preferred  to  provide  informal  tutorials,  with  the  SME  providing  the  information  while  the 
contractor  staff  member  took  notes.  This  method  was  less  efficient  than  a  documented  approach 
(see  the  "Lessons  Learned"  section  of  this  report),  but  was  accepted  by  the  contractor  at  the 
SMEs'  request. 

Step  3,  the  development  of  the  criterion  test  for  each  lesson,  was  generally,  at  the  request 
of  the  SMEs,  deferred  until  later  in  the  sequence,  usually  after  the  CBI  authoring  had  been 
completed  in  step  1 1 .  This  deviation  fi-om  procedure  is  also  discussed  further  in  the  Lessons 
Learned  section  of  this  report. 
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After  completion  of  their  front  end  analysis  the  SMEs  served  as  consultants  to  the  course 
designers  and  developers,  answering  questions  and  providing  additional  details  as  the  developers 
worked  with  the  information  already  received.  At  Step  8  the  standard  procedure  for  designing 
interactive,  self  paced  instruction,  would  consist  of  course  developers  preparing  storyboards, 
specifying  the  guided  practice  and  providing  the  explanation  and  demonstration  specifications 
for  use  by  the  CBI  authors. 

The  next  sections  provide  a  brief  overview  of  the  lessons:  their  overall  structure,  the  use 
of  multimedia  content,  and  the  way  in  which  student  progress  is  recorded  and  monitored. 
Appendix  B  contains  some  background  material  on  the  nature  of  multimedia  computer  based 
instruction. 

Structure  of  the  CBI  Lessons 

Each  lesson  consists  of  the  instruction  itself;  plus  a  pretest  and  a  post  test.  The  two  tests 
are  composed  of  identical  items,  but  serve  two  very  different  purposes.  The  pretest  determines 
how  much  the  trainee  already  knows  about  the  lesson  topics,  and  whether  he  or  she  is  already 
proficient  enough  to  be  allowed  to  skip  or  bypass  any  of  the  topics  of  the  lesson,  or  even  the 
entire  lesson.  The  post  test,  on  the  other  hjmd,  determines  the  trainee's  proficiencies  on  the 
lesson  topics  upon  exiting  a  lesson. 

Each  lesson  consists  of  "topics,"  which  are  subsets  of  the  objectives  that  eomprise  the 
performance  requirements  covered  by  the  lesson.  A  trainee  may,  if  desired,  go  through  the 
lesson  in  any  desired  sequence  of  topics,  although  trainees  new  to  the  subject  matter  are  advised 
to  proceed  in  the  sequence  specified  by  the  course  map. 

The  intent  of  CBI  is  to  provide  guided  practice,  keeping  the  trainee  actively  involved  with 
the  subject  matter:  interpreting  information,  processing  it,  and  applying  it  to  answer  questions 
and  perform  tasks  presented  in  the  lessons.  In  these  lessons  guided  practice  takes  the  form  of 
questions  and  tasks  presented  to  the  trainee  approximately  every  two  to  four  minutes.  The 
trainee  receives  immediate  feedback  after  each  response;  that  is,  the  trainee  is  told  whether  a 
given  answer  was  correct  or  incorrect.  The  corrective  function  of  feedback  after  an  error  is  to 
keep  the  trainee  from  forming  a  misinterpretation  or  faulty  conclusion  and  carrying  it  forward, 
thinking  that  it  is  correct. 

Multimedia  Components  of  the  CBI  Lessons 

The  computers  used  by  the  trainees  are  essentially  standard,  relatively  high  performance 
(80486-based)  systems  with  multimedia  kits  (CD  ROM  drive,  sound  card,  speakers).  Additional 
capabilities  consist  of  a  local  area  network  (LAN)  card,  a  fax/modem  card,  and  a  special  video 
card.  The  soimd  and  video  cards  allow  IconAuthor  to  present  audio  files  and  video  files  as  part 
of  the  lessons;  the  fax/modem  card  and  the  LAN  card  allow  the  CBI  computers  to  communicate 
with  each  other,  locally  or  at  a  distance  over  phone  lines. 
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The  CBI  authors  used  the  IconAuthor  course  writing  software  to  design  lessons  that 
present  screens  containing  text,  still  or  animated  graphics,  video  stills  or  animated  clips,  and 
audio  clips,  either  alone  or  in  combination  with  each  other.  Audio  files  accompanying  other 
media  were  either  started  automatically,  or  were  controlled  by  the  trainee  through  the  use  of  an 
‘audio  button’  that  allowed  the  trainee  to  activate  the  audio  when  he/she  was  ready  for  it,  and 
allowed  the  audio  to  be  repeated  as  often  as  the  trainee  desired.  Each  lesson  made  use  of  all  of 
these  components,  as  required. 

A  variety  of  question  and  answer  forms  appear  in  the  lessons,  mostly  multiple  choice, 
true  false,  matching,  and  sequencing,  with  occasional  requirements  for  the  trainee  to  type  in  short 
entries.  Some  of  the  questions  were  about  verbal  descriptions  of  events,  conditions,  or  scenarios; 
as  often  as  was  possible  and  practical,  the  questions  were  about  graphic  representations  of  those 
events,  conditions,  or  scenarios. 

Because  it  was  clear  that  many  of  the  soldiers  making  up  the  target  audience  for  the  lane 
lessons  might  have  had  no  previous  experience  with  computers,  or  with  Windows'  features  such 
as  the  graphical  interface,  the  mouse,  dialogue  boxes,  buttons,  and  so  on,  an  introduction  was 
prepared,  presented  by  the  computer,  to  explain  and  demonstrate  to  the  trainee  just  how  the 
lessons  and  the  navigational  elements  were  to  be  used.  This  introductory  "lesson"  was 
informally  beta  tested  and  delivered  as  an  element  included  at  the  beginning  or  each  lane  CD 
ROM  to  be  delivered  to  the  field. 

Test  Performance  Criteria  and  Course  Progress 

The  trainee’s  raw  score  on  the  quiz  at  the  end  of  a  topic  is  compared  to  a  pass/fail 
criterion  established  by  the  SME  and  instructional  designer  and  verified  by  the  external  SMEs 
who  reviewed  the  lessons.  Generally  those  internal  topic  criteria  range  between  80  and  100 
percent.  The  trainee  is  told  only  that  he  or  she  has  passed  or  failed  the  quiz. 

When  the  trainee  completes  the  topic  quiz,  the  lesson  returns  to  the  lesson  selection  menu 
screen.  If  the  topic  has  been  completed  with  a  test  score  above  the  established  criterion  score,  a 
"thumbs  up"  graphic  appears  beside  the  button  for  that  topic,  indicating  that  the  trainee  has 
received  a  "go,"  or  passing  score.  If  the  score  was  below  the  required  level,  the  thumb  points 
down.  The  trainee  is  advised,  whenever  a  “thumbs  down”  appears,  that  he  or  she  should  retake 
that  topic  before  taking  the  post  test.  Full  credit  is  recorded  for  a  trainee  for  a  lesson  only  if  the 
post  test  indicates  that  all  topics  have  been  passed  successfully. 

It  should  be  noted  that  although  the  trainee  receives  immediate  feedback  after  all  active 
responses  during  the  lesson  itself,  no  feedback  is  provided  after  each  of  the  test  items  on  either 
the  pretest  or  the  post  test.  Providing  feedback  on  the  test  questions  might  tempt  trainees  to 
record  the  answers  to  the  questions  and  then  use  them  to  earn  undeserved  credit. 
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Formative  Evaluation  of  Lessons 


A  formative  evaluation  plan  was  prepared,  providing  methods  and  procedures  for  quality 
review  of  the  CBI  lessons  at  several  stages  (Deterline,  1994).  The  QA  conducted  at  steps  5  and  6 
of  the  developmental  sequence  (Table  3)  involved  CBI  developers  and  SMEs  working  on  other 
lessons.  The  basic  purpose  of  this  review  was  to  get  an  independent  review  of  the  initial  design 
and  development  specifications  for  the  lesson. 

The  alpha  tests  (step  12  of  Table  3)  were  quality  control  reviews  of  completed  CBI  by 
project  SMEs  other  than  those  assigned  to  that  lane,  as  well  as  other  SMEs  from  Camp  Dodge 
units.  For  example,  the  commanders  and  selected  staff  of  the  Regional  Training  Site  — 
Maintenance,  and  the  Equipment  Maintenance  Center  --  CONUS,  located  at  Camp  Dodge, 
reviewed  several  of  the  maintenance  lessons  for  technical  accuracy  and  to  provide  comments  on 
the  instructional  design  of  the  lessons.  Similarly,  members  of  a  medical  company  based  at  Camp 
Dodge  reviewed  the  medical  lessons.  Other  appropriate  SMEs  reviewed  the  other  lane  lessons. 
Generally,  the  number  of  problems  navigating  through  the  course  exceeded  the  number  of 
doctrinal  errors.  All  errors  or  flaws  found  during  alpha  testing  were  corrected  (step  13)  before  a 
lesson  went  on  to  be  beta  tested. 

The  beta  testing  (step  14)  was  a  direct  test  of  the  effectiveness  of  each  lesson  on  target 
audience  trainees.  Soldiers  holding  the  MOSs  for  which  the  lane  lessons  were  designed  (when 
such  trainees  were  available  and  accessible),  took  each  lesson  in  exactly  the  manner  in  which  the 
lessons  will  be  taken  in  the  field.  Each  lesson  was  beta  tested  by  a  minimum  of  two  test  trainees, 
with  revisions  being  made  after  each  test,  as  necessary  (steps  15  and  16).  As  a  rule,  if  revisions 
were  required  after  the  second  test  trainee,  an  additional  trainee  beta  tested  the  lesson.  Keesling 
(1995)  documents  the  beta  test  results  on  the  lane  training  CBI  lessons,  showing  that  they  are 
very  effective  at  training  the  target  students. 

STAFF  LESSONS 

Concurrent  with  the  project  to  develop  computer-based  training  for  FSB  personnel, 

ARPA  initiated  a  project  conducted  at  Ft.  Beiming,  GA  to  develop  training  for  the  battle  staff  of 
a  maneuver  battalion.  Lessons  developed  in  that  project  were  the  basis  for  the  development  of 
lessons  for  the  staff  of  the  FSB,  and  its  counterpart  in  the  separate  brigade  —  the  Support 
Battalion  (SB).  This  section  of  the  report  describes  the  staff  positions  for  which  instruction  was 
to  be  developed,  outlines  the  Battle  Staff  Training  System  (BSTS)  work  at  Ft.  Benning,  and 
describes  how  the  FSB  and  SB  staff  lessons  were  developed  in  coordination  with  that  project. 

FSB  and  SB  Staff  Positions  Compared  to  Maneuver  Battalion  Staff  Positions 

The  staff  structures  of  the  FSB  and  SB  differ  from  each  other  and  from  that  of  the 
maneuver  battalion.  These  discrepancies  had  to  be  accounted  for  in  developing  the  instruction 
for  the  different  staffs.  Some  of  the  staff  officers  perform  essentially  the  same  functions: 
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•  The  Executive  Officer  PCO)  in  all  battalions  is  devoted  to  staff  supervision. 

•  The  SI  is  devoted  to  personnel. 

•  The  S4  manages  internal  logistics. 

The  major  differences  are  associated  with  the  S2  and  S3  positions.  In  addition,  the  FSB  and  SB 
have  additional  staff  officers  not  present  in  maneuver  battalions.  These  differences  are 
highlighted  in  Table  4. 

The  FSB  staff  structure  combines  the  S2  and  S3  positions  and  devotes  this  position  to 
internal  management.  The  S2  of  the  SB  also  is  internally  focused.  In  both  cases  these  officers 
do  not  have  the  reconnaissance  asset  (scout  platoon)  that  the  maneuver  battalion  S2  must 
manage.  The  S2/3  of  the  FSB  and  the  S2  of  the  SB  are  concerned  with  security  of  the  area(s) 
from  which  support  services  are  provided,  they  coordinate  with  other  staffs  to  obtain  the 
intelligence  they  need  to  develop  security  plans. 

The  FSB  structure  adds  the  Support  Operations  Officer  (SPO)  to  manage  the  external 
relationship  to  the  supported  unit(s).  The  SB  structure  gives  those  duties  to  the  S3.  In  a 
maneuver  battalion  the  S3  is  directs  combat  operations  in  accord  with  the  Commander’s  intent. 
The  SB  structure  adds  a  Brigade  Material  Management  Officer  (BMMO)  to  provide  additional 
management  of  supply  and  maintenance  functions. 

These  differences,  which  are  only  sketched  here,  meant  that  the  instruction  for  the  FSB 
staff  officers  had  to  be  modified  considerably  from  the  materials  originally  developed  for 
maneuver  battalion  staff  officers.  Instruction  for  the  SPO  and  BMMO  was  developed  entirely  by 
the  Camp  Dodge  team  of  SMEs,  instructional  developers,  and  CBI  authors. 

Appendix  C  gives  the  original  course  content  specification  for  the  FSB  and  SB  staff 
positions,  as  developed  by  the  Camp  Dodge  SMEs.  This  appendix  also  indicates  the  estimated 
instructional  time  for  these  lessons.  The  BMMO  course  was  designed  to  be  used  in  either  of  two 
ways,  depending  on  the  prior  experience  of  the  each  trainee:  trainees  "with  previous  experience  as 
an  SPO  in  an  FSB  need  take  only  the  lessons  in  the  BMMO  course.  If,  however,  the  trainee 
lacks  that  prior  experience,  both  the  SPO  course  and  the  BMMO  course  will  be  assigned. 
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Table  4 


Comparison  of  selected  staff  positions:  FSB,  SB  and  maneuver  battalion 


Maneuver  Battalion 

Forward  Support  Battalion 

Support  Battalion 

staff 

Position 

Responsibilities 

staff 

Position 

Responsibilities 

staff 

Position 

Responsibilities 

S2 

Internal:  Intelligence 
(Enemy,  Terrain, 
Security,  Weather); 
Combat  Area 

S2/S3 

Internal  Operations: 

Operations, 

Intelligence, 

Security,  Training 

S2 

Internal: 

Intelligence 
(Enemy,  Terrain, 
Security,  Weather); 
Support  Area 

S3 

Internal:  Combat 
Operations;  Training 

S3 

External:  Manages 
Supply, 

Maintenance  and 
Medical  Companies 
through  three 
assistant  S3s. 

SPO: 

Support 

Operations 

Officer 

External:  Manages 
Supply, 

Maintenance  and 
Medical  Companies; 
coordinates 
transport  and  field 
services. 

BMMO: 

Brigade 

Material 

Manage¬ 

ment 

Officer 

Manages  Supply 
and  Maintenance; 
Controls  Property 
Book 

The  Battalion  Staff  Training  System  (BSTS)  Project  at  Ft.  Beiming. 

In  addition,  to  the  staff  members  discussed  above,  the  BSTS  project  also  developed 
lessons  for  the  Fire  Support  Officer,  Air  Defense  Officer,  S3-Air,  Engineer  Officer,  Signal 
Officer  and  Chemical  Officer. 

The  staff  lessons  developed  at  Ft.  Benning  were,  generally,  of  a  different  character  than 
the  interactive  CBI  lessons  developed  for  the  lanes  training  of  the  operational  companies  of  the 
FSB,  discussed  previously.  These  lessons  made  use  of  a  technique  referred  to  as  Computer 
Managed  Instruction  (CMI)  that  serves  to  tie  off-line  study  to  computer-based  testing  and  record 
keeping.  Typically,  each  lesson  begins  with  a  brief  introduction  at  the  computer,  followed  by  the 
lesson  pretest.  The  on-screen  directions  then  tell  the  trainee  to  leave  the  computer  to  read  a  text 
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lesson^,  then  return  to  the  computer.  Upon  the  trainee's  return,  the  computer  then  provides  a 
practical  exercise,  followed  by  the  post  test  for  the  lesson.  The  trainee  receives  go/no-go 
information  on  the  lesson  topics,  with  advice  to  restudy  the  no-go  topics.  Then  the  CMI  track 
moves  to  the  next  lesson  and  a  similar  set  of  components. 

CMI  allows  integrated  use  of  any  variety  of  off-line  instructional  materials,  media,  or 
instructional  events.  These  instructional  events  might  include  activities  such  as:  taking  CBI 
lessons  that  are  not  otherwise  integrated  into  the  training  management  system;  reading  text 
materials;  looking  up  information  in  reference  documents;  watching  a  videotape;  participating  in 
a  simulator  exercise;  observing  a  demonstration  of  equipment;  participating  in  or  observing  a 
field  exercise  and  critique;  and  any  other  kind  of  instructional  or  informational  event  deemed 
relevant  to  the  course  objectives. 

Coordination  of  the  Ft.  Benning  and  Camp  Dodge  Projects 

Completed  products  from  Ft.  Benning  were  to  feed  into  the  development  of  the  FSB  staff 
lessons  at  Camp  Dodge.  The  plan  was  for  the  SME  and  contractor  staff  in  Iowa  to  receive  the 
Ft.  Benning  lessons  as  those  lesson  were  completed,  and  modify  them  for  the  FSB  and  SB  staffs. 
The  plan  specified  that  this  would  involve  removing  lessons  or  segments  of  lessons  that  were  not 
relevant  to  the  performance  of  the  CSS  FSB  and  SB  staffs,  and  adding  CSS  specific  topics. 

This  plan  assumed  that  all  lessons  would  be  CBI  lessons.  When  a  lesson  developed  at 
Ft.  Benning  and  sent  to  Camp  Dodge  was  in  CBI  form,  the  modification  for  the  CSS  staffs  was 
relatively  straightforward.  These  revisions  involved  removing  battle  staff  content  that  was  not 
relevant  to  the  FSB  staffs  (in  some  cases,  deleting  entire  lessons  for  this  same  reason),  and 
adding  segments  to  cover  tasks  unique  to  the  FSB  or  SB,  not  covered  in  the  Ft.  Benning 
materials. 

The  text  lessons  presented  a  more  difficult  problem.  One  possibility  would  have  been  to 
convert  the  text  lessons  into  CBI,  making  the  appropriate  changes  during  this  revision.  This 
would  have  been  tantamount  to  developing  all  new  CBI  instruction,  taking  very  little  advantage 
of  the  development  at  Ft.  Benning.  The  contract  did  not  have  a  provision  for  such  a  large 
undertaking.  The  alternative  chosen  was  to  modify  the  text  lessons  and  the  accompanying  CMI. 
In  this  way  the  staff  lessons  for  the  maneuver  battalion  and  the  FSB/SB  have  the  same  Took  and 
feel.’  Using  additional  funding  provided  by  ARP  A,  the  projects  at  Ft.  Benning  and  Camp  Dodge 
hired  additional  personnel  to  accomplish  the  conversion  of  the  text  lessons. 


^  A  text  or  print  lesson  is  hardcopy  developed  by  the  Ft.  Benning  staff,  gathering  and 
rewriting  information  from  training  manuals,  field  manuals,  and  other  relevant  source 
documents. 
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The  Camp  Dodge  staff  revised  and  modified  the  CMI  segments  and  practical  exercises 
for  all  of  the  FSB  and  SB  text  lessons.  All  of  the  CMI  segments  had  to  be  revised  because  the 
introductory  audio  narrative  and  print  screens  addressed  battle  staff  topics  that  was  inappropriate 
for  the  FSB/SB  staffs.  This  meant  modifications  had  to  made  to  the  introductions,  exercises,  and 
to  the  pretests  and  post  tests. 

Camp  Dodge  staff  performed  the  tasks  involved  in  combining  relevant  portions  of  the  S2 
and  S3  SB  lessons  developed  at  Ft.  Benning  to  create  the  S2/3  text  lessons,  CBI  lessons  and 
CMI.  In  early  June  of  1995,  with  the  work  appropriately  50%  complete  on  the  S2/3  lessons, 
word  was  received  from  ARPA  that  the  S2/3  materials  need  not  be  completed. 

Appendix  E  describes  the  formative  evaluation  of  the  FSB/SB  lessons. 

THE  TRAINING  MANAGEMENT  SYSTEM:  EMMii 

It  is  not  enough  to  provide  training  for  soldiers.  Even  very  effective  training  that  helps 
soldiers  develop  mastery  of  relevant  tasks  that  are  essential  to  performance  of  unit  missions  must 
be  managed  so  that  it  can  be  made  available  to  trainees  on  a  timely  basis,  and  must  yield 
information  that  commanders  at  all  levels  can  use  to  determine  the  readiness  levels  of  individual 
soldiers  and  units. 

Purpose 

In  both  public  education  and  military  and  industrial  training,  it  has  long  been  known  that 
"completion  of  a  course,"  or  "exposure  to  n  hours  of  instruction"  are  less  reliable  indieators  of 
individual  competencies  than  direct  assessments  of  performance.  The  military  emphasis  on 
training  to  attain  a  specified  level  of  proficiency  on  mission  essential  tasks  reflects  the 
understanding  that  participation  in  a  number  of  exercises  does  not  guarantee  unit  readiness.  Unit 
commanders  at  all  levels  must  have  access  to  information  about  individual  and  unit  proficiency 
so  that  they  may  allocate  training  resources  to  allow  their  units  to  attain  and  sustain  their 
assigned  readiness  status. 

The  requirements  of  any  training  management  system  are  that  individual  and  unit 
performance  on  relevant  tasks  must  be  observed,  measured  and  evaluated,  identifying 
shortcomings  as  well  as  competencies.  The  measurements  and  evaluations  must  be  recorded  and 
made  readily  accessible  to  commanders  and  training  managers  so  that  they  may  determine  which 
units  and  soldiers  need  what  additional  training.  The  management  system  should  also  contain 
information  on  what  training  is  available,  and  tools  the  commander  and  training  managers  can 
use  to  manage  unit  training  to  eliminate  any  critical  deficiencies  in  the  performance  of 
individuals  and  units. 
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The  Environment  for  Multi-Media  Interactive  Instruction 


The  Environment  for  Multi  Media  Interactive  Instruction  (EMMii)  was  developed  by 
BDM  from  its  proprietary  Training  Management  System,  a  system  that  was  originally  developed 
in  an  industrial  setting  to  provide  the  fiinctions  outlined  above.  EMMii  is  a  shell  that  surrounds 
the  instruction,  providing  controlled  access  and  egress  so  that  the  information  needed  by  training 
managers  may  be  gathered  automatically  as  the  soldiers  progress  through  the  lessons.  The 
functions  and  applications  that  this  system  permits  are  outlined  below.  Additional  information 
may  be  obtained  from  the  User  Manual  (BDM,  1995). 

The  EMMii  introduces  the  trainee  to  a  set  of  lessons,  matches  the  records  to  the  soldier's 
name.  Social  Security  number,  unit  and  other  pro  forma  information,  then  records  pretest  and 
post  test  scores  and  the  designations  of  the  lessons  completed  successfully  or  completed  only  in 
part.  The  data  gathered  from  trainee  lesson  performance  is  stored  in  data  bases  that  can  be 
accessed  by  individuals  identified  for  EMMii  by  a  System  Administrator  (SA).  The  SA,  the 
highest  level  of  EMMii  manager,  can  specify  various  level  of  access,  so  that  any  desired  level  of 
confidentiality  and  security  can  be  established  and  maintained.  The  trainees,  for  example,  have 
‘read  only’  access  to  lessons  and  to  their  own  records:  they  may  not  edit  any  entries  pertaining  to 
their  schedules  or  performance  data. 

The  Trainer's  and  Student  Guides  to  the  lessons  and  EMMii  (xxx  references  xxx)  contain 
directions  for  properly  accessing  EMMii;  the  Student  Guide  also  contains  a  job  aid  to  guide  a 
trainee  through  EMMii  and  the  selection  of  lessons,  and  directions  on  how  to  transfer  the 
performance  data  back  to  the  training  manager.  Along  with  these  two  Guides,  units  will  also 
reeeive  a  copy  of  the  EMMii  User  Manual  for  use  by  the  SA  and  by  training  managers. 

Applications 

EMMii  can  deliver  lessons  in  three  basic  ways  (with  variations): 

•  via  a  local  area  network  (LAN)  of  two  or  more  computers,  with  a  manager  station  that 
maintains  the  master  and  subordinate  databases  and  controls  a  file  server 

•  via  student  stations  in  standalone  configurations  tied  to  a  manager  station  via  modem 

•  via  CD  to  standalone  student  stations  that  are  not  connected  to  any  other  computers. 

In  the  latter  application,  the  standalone  student  station  could  be  located  in  an  armory,  some  other 
location  designated  for  ARNG  training,  or  at  a  location  most  facilitative  to  the  trainee,  such  as 
the  trainee's  residence. 

Typically,  the  LAN  setup  would  have  all  computers  in  a  single  location  (e.g.  an  armory). 
In  this  setting  the  student  would  not  use  lessons  on  CDS.  The  training  manager  would  determine 
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a  trainee's  schedule  in  consultation  with  the  trainee's  commander  and  the  trainee  to  be  sure  that 
times  and  dates  are  viable.  Then  the  training  manager  would  schedule  individual  lessons  for  a 
trainee,  assigning  a  specific  student  station  by  date  and  time.  EMMii  would  determine  the 
appropriate  time  to  begin  dovraloading  the  scheduled  lesson  from  the  file  server  into  the 
identified  student  station  for  the  trainee.  Because  multimedia  CBI  lessons  tend  to  be  large  in  size 
(tens  of  megabytes)  the  downloading  time  is  significant  and  downloading  must  be  scheduled  for 
blocks  of  time  when  the  file  server  and  student  station  can  be  unavailable  for  other  uses.  When 
the  trainee  arrives  at  the  scheduled  time,  the  lesson  is  ready,  the  student  accesses  the  lesson  and 
completes  it.  Student  times  for  beginning  the  lesson,  completing  it,  and  test  scores  are  gathered 
by  EMMii  and  stored  in  the  appropriate  database  in  the  manager  station.  Depending  on  what  is 
scheduled  next  for  that  student  station,  EMMii  will  then  unload  the  completed  lesson. 

In  an  armory  without  an  established  LAN,  a  trainee  Avill  pick  up  a  lesson  CD  and  a  3.5" 
floppy  disk  from  the  training  manager  or  a  designate  performing  that  function.  Following  the 
directions  in  the  Student  Guide,  the  trainee  will  turn  on  the  computer,  which  will  boot  to  the 
EMMii's  trainee  selection  screen.  The  trainee  enters  a  user  ID  and  password;  selects  a  lesson, 
then  inserts  the  floppy  disk  and  proceeds  through  the  lesson  and  its  tests.  After  the  trainee 
completes  the  lesson,  the  EMMii  will  load  the  lesson  data  onto  the  floppy  disk.  The  disk  is  then 
returned  to  the  training  manager  or  representative,  who  will  copy  the  data  from  the  floppy  disk 
into  the  appropriate  database. 

In  some  settings  trainees  will  be  allowed  to  take  a  student  station  home  to  take  the 
lessons  there.  The  course  CDS  will  usually  be  provided  to  the  trainee  at  the  time  the  computer  is 
signed  out.  The  floppy  disk  that  will  serve  as  the  interface  between  the  training  manager's  station 
and  the  remote  location  will  also  be  delivered  to  the  trainee  at  that  time.  The  trainee  will  use  the 
CD  and  the  floppy  disk  exactly  as  they  would  be  used  in  the  non-LAN  armory.  To  expedite 
transmission  of  the  trainee’s  data  to  the  EMMii,  instead  of  hand  delivering  the  floppy  disk  and  its 
data  to  the  training  manager,  the  trainee,  upon  completing  the  lesson,  will  coimect  the  computer's 
modem  to  the  home  telephone  line,  and  the  data  will  be  sent  from  the  student  station  to  the 
manager  station  via  modem. 

Commanders  or  their  representative  can  access  the  lesson  databases  from  various  points 
of  view.  One  might  check  on  individual  soldiers  to  determine  which  lessons  they  completed  and 
when.  Or  one  might  want  to  determine  how  many  commissioned  officers  or  senior 
noncommissioned  officers  have  successfully  completed,  say,  the  Defend  Sector  or  the  staff  S2 
course,  or  which  lessons  from  those  courses.  Such  searches  can  also  be  by  sub-imits:  a  battalion 
can  check  on  each  of  its  companies  and  the  platoons  within  each  company  separately.  In  this 
way,  "holes"  in  the  training  of  the  units  or  of  individuals  assigned  to  the  units  can  be  determined, 
and  proper  action  can  be  taken  on  a  timely  basis. 

EMMii  also  keeps  track  of  the  responses  made  to  each  of  the  quiz  and  test  items  on  each 
lesson.  Over  time,  masses  of  performance  data  will  be  accumulated  for  large  numbers  of 
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soldiers.  EMMii  also  accumulates  opinion  statements  from  trainees.  For  this  purpose  the  trainee 
is  offered  a  choice  from  among  statements  such  as: 

1 .  The  vocabulary  used  in  this  lesson  is  about  right  for  me. 

2.  The  vocabulary  used  in  this  lesson  was  too  simple. 

3.  The  vocabulary  used  in  this  lesson  was  too  complicated. 

EMMii  accumulates  counts  of  such  selections,  and  these,  like  the  item  response  counts, 
can  be  accessed  by  CBI  authors  and  instructional  developers  for  the  purpose  of  identifying  areas 
needing  revision  or  improvement. 

EMMii  is  not  a  simple  system  to  learn  or  use.  In  order  to  utilize  it  to  its  full  extent, 
individuals  with  computer  experience  will  have  to  spend  considerable  time  mastering  the  system. 
The  need  for  training  in  the  use  of  this  system  led  the  ARNG  and  ARPA  to  substitute 
development  of  such  training  for  the  collective  training  lanes  mentioned  earlier  (Table  2). 

LESSONS  LEARNED 

This  section  of  the  final  report  presents  observations  about  the  project  that  could  prove 
useful  to  others  attempting  similar  developmental  efforts.  These  observations  are  organized  into 
three  themes: 

Staff  Resources 

Hardware 

CBI  Production  Methodology 

Project  Expectations  and  Oversight 

In  some  cases  the  lesson  learned  involves  more  than  one  of  the  theme  areas.  Rather  than  repeat 
the  lessons,  they  are  placed  where  they  seem  most  relevant. 

Staff  Resources 

ARPA/IANG  .staffing.  ARPA  and  the  lANG  were  very  generous  in  providing  SMEs  to 
support  the  project.  These  SMEs  were  very  proficient  in  their  areas  of  expertise,  and  they  had 
prior  experience  with  managing  and  delivering  Army  training.  They  were  less  experienced  with 
such  Army  training  models  as  instructional  system  design  (ISD)  and  had  virtually  no  prior 
experience  with  developing  CBI.  A  few  had  experience  with  the  lANG  lanes  training 
methodology  which  parallels  the  ISD  model.  The  lack  of  training  in  ISD  meant  that  they  did  not 
find  it  easy  to  develop  terminal  learning  objectives  and  work  backwards  to  identify  all  the 
enabling  objectives  that  would  require  training.  The  inexperience  with  developing  CBI  left  them 
vmsure  as  to  how  to  chose  among  media  and  methods  for  presenting  a  particular  topic,  and  what 
capabilities  they  could  expect  of  the  hardware  and  software.  To  some  extent,  the  expertise  of  the 
contractor  staff  could  make  up  for  the  SMEs  inexperience  with  multimedia  instructioniil 
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development,  but  the  contractors  could  not  substitute  for  the  SME  expertise  in  identifying 
learning  objectives  and  measures  of  proficiency. 

Contractor  staffing.  The  initial  staffing  by  the  contractor  consisted  of  a  manager  and 
administrative  assistant,  and  four  people  who  were  to  design  and  author  the  courses.  Each  was 
paired  with  an  SME  to  work  on  one  of  the  training  lanes.  These  personnel  had  academic 
backgrounds  in  instructional  design  and  CBI,  and  most  of  them  had  worked  on  prior  CBI 
development  projects.  None  of  them  had  prior  experience  with  the  software  system  chosen  for 
this  project  (IconAuthor).  The  contractor  staff  was  given  only  minimum  training  in  December, 
1993,  on  the  authoring  system.  Two  people  attended  a  one- week  workshop,  while  three  other 
staff  members  were  given  two  days  of  training. 

The  inexperience  with  IconAuthor  meant  that  while  they  were  trying  to  understand  the 
subject  matter  sufficiently  to  design  a  logical  course  flow,  including  all  the  guided  practice,  the 
instructional  designer/authors  were  also  trying  to  learn  enough  about  the  features  of  the  software 
to  be  comfortable  with  making  choices  about  the  way  in  which  to  present  the  material.  Some  of 
the  capabilities  of  IconAuthor  required  a  facility  with  the  C  programming  language,  and  only  one 
of  the  contractor  staff  had  such  experience. 

Later  in  the  project  it  became  clear  that  additional  persoimel  would  have  to  be  hired  to 
meet  the  deadlines  and  that  greater  role  differentiation  would  also  be  beneficial.  Additional 
personnel  were  hired  and  contractor  personnel  were  assigned  either  to  design  courses  (i.e. 
develop  storyboards  and  identify  supporting  materials),  or  to  create  the  comses  using 
IconAuthor''.  Had  it  been  possible  to  hire  some  IconAuthor  experts  at  the  outset  (none  of  the 
applicants  had  this  expertise)  this  role  differentiation  could  have  been  established  earlier  which 
would  have  facilitated  the  production  of  the  courses. 

Another  example  of  the  need  for  role  differentiation  was  the  early  lack  of  personnel 
devoted  to  the  creation  of  graphic  materials.  Project  managers  anticipated  being  able  to  find 
graphical  material  they  needed  in  public  domain  or  commercial  clip  art  collections.  Army 
documents,  or  archives  (photos  and  video).  This  proved  to  be  incorrect,  and  for  the  first  few 
months  of  the  project,  the  designer/authors  were  also  drawing  the  graphical  materials  they 
needed.  They  and  the  SMEs  also  found  it  necessary  to  re-shoot  video  sequences  because  the 
versions  in  the  Army  archives  were  not  of  sufficient  quality  to  import  into  the  lessons.  Later  in 
the  project,  full  motion  video  was  used  less  frequently  and  graphic  artists  were  hired  to  create 
drawings  and  animations. 

Recommendations.  Projects  such  as  this  should  try  as  much  as  possible  to  maintain  the 
role  distinctions  between  SMEs,  instructional  designers/developers,  and  CBI  authors.  Each  of 


'*  It  is  worth  noting  that  the  BSTS  project  developed  storyboards  in  one  location  while 
IconAuthor  experts,  in  another  location,  converted  them  into  covnses. 


17 


these  three  groups  will  benefit  from  knowing  exactly  what  each  of  the  other  groups  requires  as 
inputs,  what  each  group  does  and  produces  as  outputs,  and  how  the  work  of  each  group  can  most 
advantageously  and  beneficially  serve  the  other  two  groups.  Training  can  be  provided  to 
increase  this  understanding  at  the  outset. 

In  the  case  of  the  present  project,  the  SMEs  and  some  contractor  personnel  would  have 
benefited  from  training  in  the  ISD  methodologies.  The  ISD  training  can  be  lighter  for  the  CBI 
authors,  but  should  be  thorough  for  SMEs  and  the  instructional  developers.  The  contractor  staff 
(designers  and  authors)  would  have  benefited  from  training  in  military  culture  (e.g.  terminology, 
equipment,  operating  procedures),  focusing  on  the  units  for  which  they  were  developing 
instruction.  Additional  training  in  the  instructional  authoring  software  (in  this  case,  IconAuthor) 
would  also  have  been  beneficial.  Introductory  training  (stressing  capability  and  functionality) 
would  have  sufficed  for  the  SMEs  and  developers,  while  more  thorough  training  (stressing 
application  and  development)  would  be  required  for  CBI  authors.  The  ISD  training  could  have 
been  provided  by  contractor  personnel  (perhaps  drawing  on  other  corporate  resources),  while  the 
SMEs  could  have  provided  the  military  background  training. 

Hardware 

Desktop  and  portable  computers  were  purchased  to  perform  three  different  functions: 
Authoring  courses;  delivering  instruction  to  students;  and  providing  general  support  to  the 
project.  The  latter  machines  were  called  ‘administrative’  computers.  Eight  such  machines  were 
purchased  so  that  the  SMEs  and  contractor  staff  would  have  the  capability  of  preparing  text  and 
drawings,  as  needed.  Some  of  these  computers  were  also  used  for  project  planning  and 
administration.  These  systems  were  made  available  in  the  early  fall  of  1993,  after  the  staff  had 
been  assembled. 

In  November  1993,  two  multimedia  authoring  computers  were  provided  to  the  project. 
Each  was  a  486,  66  MHZ  computer  with  eight  megabytes  of  memory,  a  540  megabyte  hard 
drive,  1 .4  megabyte  floppy  drive,  CD  ROM  drive,  with  a  fax/modem  card  and  audio  and  video 
cards,  and  a  Super  VGA  color  14"  monitor,  101  key  keyboard,  and  a  mouse.  Each  authoring 
station  also  had  an  installed  tape  backup  unit  and  an  uninterruptable  power  supply  and  surge 
protector.  Purchased  as  ‘high-end’  machines,  by  present  (1995)  standards  these  computers  are 
barely  ‘entry  level’  for  serious  applications. 

Problems  in  determining  how  to  configure  the  authoring  system  computers,  the  software, 
and  the  add-in  cards  to  prevent  conflicts  that  caused  the  systems  to  crash  introduced  delays  in  the 
project  during  November  and  December  of  1993.  One  of  these  machines  proved  to  be 
persistently  unreliable,  despite  several  service  visits  and  one  return  trip  to  the  manufacturer.  The 
late  and  constrained  availability  of  these  machines  delayed  the  training  in  IconAuthor,  and 
created  a  bottleneck  in  development  as  the  four  authors  were  often  contending  for  the  machines. 
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Several  hardware  problems  were  related  to  the  desire  to  provide  high-quality  video  elips. 
Capture  and  playback  of  high  quality  video  clips  required  a  special-purpose  video  card.  There 
were  persistent  problems  in  configuring  the  administrative  machines  (being  used  for  draft 
development)  and  the  authoring  stations  to  use  the  same  color  palettes.  Thus,  color  choices  (e.g. 
for  graphics)  would  not  remain  consistent  when  ported  between  the  administrative  machines  and 
the  authoring  stations.  Sometimes  this  was  merely  annoying  (or  amusing),  other  times  it  meant 
that  information  on  the  screen  could  not  be  seen.  This  problem  was  resolved  by  standardizing  on 
the  authoring  stations  as  the  ‘target’  platform.  To  the  extent  possible,  the  administrative  and 
student  stations  were  configured  to  work  properly  with  this  standard. 

The  special  video  boards  created  another  problem.  When  additional  authoring  machines 
were  rented  to  support  the  staff  added  later  in  the  project,  these  video  cards  were  no  longer 
available.  This  meant  that  the  bottleneck  could  only  be  partially  relieved  by  acquiring  the  added 
equipment. 

Home  quality  VCRs  proved  to  be  inadequate  for  copying  video  from  tapes  provided  by 
the  government,  so  the  contractor  rented  a  professional  quality  video  camera  with  a  high  quality 
playback  unit.  The  camera  was  also  used  to  record  video  footage  when  that  could  be  taken  at 
Camp  Dodge  or  nearby  armories.  The  rental  cost  eventually  added  up  to  more  than  it  would 
have  cost  to  purchase  this  equipment. 

Video  is  costly  in  other  ways.  Video  clips  take  up  a  lot  of  disk  space  (several  megabytes 
for  one  minute  of  video  is  not  atypical),  increasing  the  storage  requirements  for  the  course.  The 
size  of  these  files  also  means  that  students  are  required  to  wait  some  time  for  the  file  to  load 
before  it  can  be  played.  These  delays  are  sometimes  quite  distracting. 

Video  on  computer  is  still  an  emerging  technology.  It  tends  to  be  jerky  and  to  lose 
resolution.  Some  of  the  medical  lessons  benefited  from  the  use  of  video  to  show  exactly  how  to 
carry  out  certain  procedures  with  high  realism.  These  videos  had  been  prepared  expressly  for  the 
purpose  of  illustrating  these  procedures;  extant  video  is  not  often  aimed  at  the  desired 
instructional  objectives.  Otherwise,  this  technology  was  not  generally  required.  The  videos  that 
were  used  as  introductions  or  attention  getters  (trainees  who  beta-tested  the  lessons  did  express 
appreciation  for  these  video  clips)  did  not  have  to  be  of  very  high  quality.  It  might  have  been 
preferable  to  use  a  lower  quality  video  capture  and  playback  system  (one  that  would  continue  to 
be  supported  by  hardware  and  software  manufacturers),  even  if  the  video  was  of  lower  quality. 

After  the  first  authoring  stations  arrived  a  "luggable"  (larger  and  heavier  than  a  typical 
laptop)  computer  was  provided.  At  the  time  this  was  considered  likely  to  be  the  kind  of 
computer  that  would  be  purchased  as  a  ‘student  station.’  The  luggable  had  a  VGA  active  matrix 
color  screen,  and  sound  and  video  cards.  The  limitations  of  the  luggable  soon  made  it  clear  that 
desk  top  computers  would  have  to  be  used  instead.  However,  the  luggable  proved  useful  and 
convenient  for  taking  lessons  to  the  lANG  Armory  where  it  could  be  connected  to  a  large  screen 
for  demonstrations  for  visitors,  and  for  use  by  the  SMEs  at  military  conferences. 
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A  black  and  white  scanner  was  purchased  for  one  of  the  administrative  computers  so 
artwork  and  other  graphics  could  be  scanned  into  the  computer.  The  scanner  had  no  Optical 
Character  Reader  (OCR)  software.  The  latter  would  have  been  helpful  when  forms,  labels,  and 
other  existing  text  had  be  scanned  in,  because  without  the  OCR,  such  material  had  to  he  scanned 
in,  in  graphic  form,  rather  than  text  form,  which  was  difficult  to  modify. 

The  black  and  white  limitation  was  not  severe  since  most  of  the  reference  material 
illustrations  (primarily  in  TMs)  were  not  in  color.  A  few  printed  color  illustrations  were 
identified  that  would  have  been  appropriate,  but  could  not  be  used.  Still  color  visuals  were 
captured  from  videos  wherever  possible  (rapid  motion  of  the  camera  or  the  subject  blurs  the 
images).  This  proved  to  be  a  very  good  source  of  color  still  images. 

Recommendations.  Each  author  should  be  provided  with  a  fully-capable  authoring 
station.  The  cost  to  do  this  is  not  prohibitive.  The  configuration  should  not  rely  on  exotic 
devices  (such  as  the  video  cards  used  in  this  project).  Standards  for  motion  video  on 
microcomputers  are  changing  rapidly.  It  is  expensive  to  shoot  new  video,  but  extant  video  is 
often  not  designed  specifically  for  the  instructional  purpose  at  hand.  Future  projects  will  need  to 
weigh  carefully  the  trade-offs  between  enhanced  quality,  costs  of  capture  and  storage,  and  the 
potential  for  continued  support  of  the  video  technology. 

Extant  video  libraries  can  be  a  valuable  source  of  still  images.  Capturing  these  images 
should  he  a  capability  of  projects  like  this.  A  color  scanner,  with  OCR  eapability,  would  also  be 
a  valuable  addition  to  such  projects,  although  a  black  and  white  scanner  may  be  sufficient. 

Removable  hard  drives  are  likely  to  be  a  better  solution  than  tape  for  back-up  and  transfer 
of  course  files.  In  some  cases  they  may  be  preferable  to  CD  ROMs,  if  the  course  is  sufficiently 
small. 


Finally,  provide  each  computer  with  the  best  surge  protection  possible.  Storms  in  the 
Camp  Dodge  area  severely  damaged  several  machines.  Uninterruptable  power  supplies  are  a 
must  for  the  authoring  stations  where  complex  work  needs  to  be  saved  and  the  machine  powered 
down  gracefully  when  the  main  power  is  disrupted. 

CBI  Production  Methodology 

In  this  section  several  techniques  for  enhancing  CBI  production  are  discussed.  An 
attempt  has  been  made  to  highlight  methods  that  may  be  isolated  from  the  hardware,  software, 
and  staff  training  lessons  learned,  to  enhance  their  range  of  application.  Topics  to  be  discussed 
are:  setting  realistic  time  frames  for  development,  following  the  developmental  sequence,  using 
standard  templates,  using  animation  in  place  of  video,  keeping  lessons  to  appropriate  length,  and 
embedding  audio  in  the  lessons. 


20 


The  first  three  of  these  lessons  learned  have  to  do  with  overall  project  management  and 
efficiencies  of  production.  They  are  somewhat  interdependent  because  following  the 
developmental  sequence  and  using  standard  templates  are  ways  to  assure  that  development  can 
be  accomplished  in  a  realistic  time  frame. 

Setting  realistic  time  frames  for  development.  Authoring  CBI  is  a  very  time  consuming 
task.  Each  lesson  requires  the  CBI  author  to  establish  the  overall  structure  of  the  lesson, 
compose  screen  copy,  create  smart  object  pages  and  variable  pages,  select  colors,  window  sizes 
and  placements,  do  the  selection  and  placement  of  active  buttons,  create  audio  files  and  video 
files,  and  provide  for  the  activation  of  audio  files  and  video  files,  graphic  files  and  embedded 
graphics.  Industry  estimates  of  time  required  to  produce  a  one  hour  highly  interactive  CBI  lesson 
range  from  300  hours  to  500  hours,  more  when  multimedia  CBI  is  the  format.  Tables  developed 
by  Golas  (1993)  indicate  that  1000  hours,  or  14  of  a  person-year,  would  be  required  (including 
time  for  development  of  storyboards)^. 

The  original  production  schedule  of  1 5  months  was  optimistic.  Floods  in  Iowa  in  the 
summer  of  1993  delayed  hiring  staff  to  some  extent  and  required  some  of  the  lANG  personnel  to 
be  involved  in  disaster  relief  operations.  The  project  authoring  stations  did  not  arrive  for  several 
months  after  the  project  start,  delaying  training  of  the  authors.  Further  delays  resulted  from 
having  to  debug  the  systems.  These  problems  had  to  be  acknowledged  in  later  adjustments  to 
delivery  schedules. 

Recommendations.  The  inadequate  early  staffing  and  the  small  nximber  of  authoring 
stations,  discussed  earlier,  are  problems  contractors  should  avoid.  On  the  government  side,  it  is 
important  to  base  schedules  of  deliverables  on  a  start  date  incorporating  the  availability  of 
hardware  and  software.  The  follow-on  to  this  project  was  funded  under  a  contract  that  states 
deliverable  due  dates  in  terms  of  months  after  the  time  when  all  hardware  and  software  are  on 
hand  and  fully  operational. 

Following  the  developmental  sequence.  In  earlier  sections  two  specific  discrepancies 
from  the  17-step  sequence  developmental  sequence  (Table  3)  were  mentioned.  These  deviations 
had  a  negative  impact  on  the  production  of  deliverables. 

1)  The  SMEs  were  to  provide  a  written  narrative  discussion  and  explanation  of  the  each 
of  the  tasks  listed  in  each  lesson's  objectives,  the  performance  setting,  the  equipment,  and 
anything  else  relevant  to  the  lesson  that  the  developers  would  need  to  know.  In  some  cases  such 
information  was  prepared  in  writing,  but  the  SMEs  generally  preferred  to  provide  necessary 
information  of  that  sort  in  conversations,  in  answering  questions,  or  after  identifying  holes  or 


^  This  estimate  assumes  that  CBI  is  to  be  used  to  make  a  medium-level  presentation 
(moderate  use  of  multimedia  components),  intended  to  teach  a  skill.  Developer  unfamiliarity 
with  target  audience  was  incorporated  as  a  factor  in  the  estimate. 
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deficiencies  in  draft  storyboards.  This  approach  did  the  job,  but  was  not  as  efficient  as  the 
written  documentation.  Furthermore,  except  for  any  notes  taken  by  the  developer,  it  did  not 
produce  adequate  documentation  for  the  quality  assurance  reviewer  to  look  at  later. 

2)  The  17  step  developmental  sequence  also  specifies  that  a  criterion  test  should  be 
developed  by  the  SMEs,  or  by  the  SMEs  and  developers  working  together,  as  the  final  step  in  the 
analysis  process.  Course  development  should  not  begin  until  the  tests  are  complete.  Many  of  the 
subsequent  design  decisions  are  thereby  made  more  easily  and  more  accurately.  The  SMEs  were 
not  familiar  enough  with  the  ISD  process  to  recognize  the  critical  importance  of  this  step,  and 
preferred  to  develop  the  tests  later.  Unfortunately,  this  resulted  in  gaps  in  training  and  testing  the 
necessary  performance  as  defined  by  the  objectives,  requiring  modifications  of  both  the  tests  and 
the  lessons. 

Another  example  of  disruption  to  the  developmental  sequence  occurred  when  the 
contractor  was  asked  to  accelerate  development  in  a  particular  lane  in  order  to  have  material 
ready  for  a  demonstration.  Despite  the  fact  that  the  contractor  pulled  staff  from  other  lanes  to 
add  to  the  target  lane,  it  was  not  possible  to  meet  this  additional  requirement.  Part  of  this  was 
due  to  the  fact  that  there  was  only  one  SME.  That  single  SME  could  only  maintain  the  same 
pace  of  instructing  developers  on  the  lesson  content.  Having  more  developers  did  not  speed  up 
the  process  because  there  was  no  way  for  them  to  work  in  parallel.  Additional  authors  would 
have  been  helpful  if  there  had  been  more  of  a  backlog  of  developed  material  to  work  with.  Of 
course  development  of  the  other  lanes  was  disrupted,  and  some  shortcuts  in  the  development  of 
the  target  lane  were  made  that  had  to  be  patched  up  later. 

Recommendations.  Training  in  the  ISD  process  would  reinforce  the  need  to  follow  the 
logical  sequence  of  developmental  steps.  The  sponsors  have  to  agree  to  the  schedule  and  be 
willing  to  show  the  intermediate  products  (storyboards,  for  example)  as  representative  of 
progress,  until  such  time  as  lessons  become  available  through  the  developmental  process. 

T  Jsing  standard  templates.  The  purpose  of  developing  or  assembling  a  standard  set  of 
templates,  or  artwork,  is  not  to  limit  the  authors,  but  to  facilitate  lesson  authoring.  Templates  or 
assembled  artwork  can  be  modified  as  desired  or  to  meet  special  requirements.  For  example,  the 
template  described  in  Appendix  B  for  navigation  buttons  was  developed  early  in  the  project  and 
used  throughout.  This  spared  the  authors  fi'om  having  to  re-invent  the  navigational  scheme  for 
each  lesson.  Similar  advantages  can  be  obtained  from  sets  of  templates  for  page  layout,  design, 
colors,  and  other  on-screen  button  functions. 

Early  in  the  project,  CBI  authors  reviewed  guidelines  and  experimented  with  various 
screen  colors  and  color  combinations,  for  backgrounds,  boxes,  buttons,  and  foreground  text. 
Standards  for  color  use  were  developed  to  provide  guidelines  for  use  in  all  lessons.  In  general, 
medium  blue  or  dark  blue  were  found  to  be  the  most  desirable  as  background  colors,  with 
highlighted  white  text.  A  second  choice  was  a  light  blue  background  with  black  text.  Green  and 
red  were  less  attractive  (except  a  very  light  green)  and  were  rarely  used,  except  where  an  outdoor 
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scene  made  green  appropriate.  In  general,  any  hue  of  dark  lettering  on  a  dark  background  proved 
to  be  undesirable  because  of  legibility  problems;  similarly,  white  lettering  on  a  light  background 
was  difficult  to  read  because  the  contrast  between  foreground  and  background  was  inadequate. 

Because  red  is  a  dramatic  color,  it  was  used  to  highlight  critical  components  of  graphic 
illustrations  or  even  critical  words  in  on-screen  text.  This  use  included  red  arrows,  circles,  red 
rectangles  with  labels  and  names  of  components,  either  fixed  in  place  on  a  graphic,  or  appearing 
briefly  and  then  vanishing.  This  technique  must  be  used  judiciously  to  avoid  wearing  out  its 
welcome. 

R emmmendations.  It  is  strongly  recommended  that  future  CBI  projects  make  maximum 
use  of  standard  templates,  developing  many  at  the  start  of  the  project,  and  adding  others,  as 
needed,  to  meet  special  requirements.  All  standard  templates  and  artwork  should  be  assembled 
into  a  convenient  hardcopy  catalog,  like  those  that  accompany  sets  of  clip  art,  showing  a  copy  of 
the  dimensions  and  layout  of  each  template  and  a  copy  of  each  piece  of  artwork. 

The  next  two  lessons  learned  concern  specific  aspects  of  multimedia  CBI.  Animation  can 
be  substituted  for  video  in  many  cases,  with  several  benefits.  An  additional  way  to  present  audio 
was  developed  in  this  project  that  may  be  useful  to  other  projects. 

Using  animation  in  place  of  video.  Animation  was  used  only  to  a  limited  degree  during 
the  early  months  of  the  project.  During  the  Spring  of  1994,  the  CBI  authors  and  the  recently 
hired  graphic  artists  began  to  explore  the  use  of  animation  as  an  effective  and  efficient  substitute 
for  motion  video,  which  was  difficult  or  impossible  to  obtain  or  produce. 

Animation  proved  especially  useful,  in  one  of  the  maintenance  lessons,  in  showing  how  a 
subsystem  was  disconnected  and  removed  from  inside  a  Bradley  Fighting  Vehicle  (BFV).  The 
trainee's  task  was  to  choose  the  next  component  that  should  be  removed,  in  its  turn.  A  line 
drawing  illustration  from  a  TM  served  as  the  basic  piece  of  artwork.  It  was  scanned  into  the 
computer,  then  retouched  and  colored  by  a  project  artist.  When  the  trainee  indicated  the  correct 
component  the  animation  showed  it  being  imbolted  and  leaving  the  scene.  For  realism,  the  soimd 
of  an  air  impact  wrench  was  introduced  as  the  retaining  bolts  were  shown  removing  themselves 
to  release  the  component.  No  mechanic  appeared  on  screen  during  the  removal  process.  That 
would  have  required  animation  of  a  human  figure  which  would  have  been  expensive  and  would 
not  have  added  to  the  instructional  value.  The  animation  was  shown  fi-om  the  point  of  view  of 
the  mechanic  as  he  or  she  would  view  the  piece  of  equipment. 

One  basic  illustration  was  used  for  the  entire  sequence,  with  drawings  of  each  component 
proportional  in  size  to  the  scene.  As  each  component  was  detached  it  was  removed  from  the 
scene.  That  component  then  was  missing  from  the  main  illustration,  having  been  touched  up  and 
out  of  the  picture.  This  was  a  very  economical  use  of  animation;  attempting  to  obtain  the  same 
scene  on  video  would  have  been  difficult  because  of  the  small  space  involved,  and  the  difficulties 
involved  in  lighting  and  photographing  the  scene  to  control  reflectance,  shadows,  and  contrast. 
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Animation  has  an  additional  advantage  that  might  in  some  cases  become  critical:  file  size. 
Even  relatively  short  video  sequences  tend  to  require  from  five  to  twenty  times  the  space  taken 
up  by  an  animation  sequence,  depending  on  the  nature  of  the  animation  and  of  the  video  used. 
The  storage  requirement  may  not  be  as  important  as  the  time  required  to  download  a  video  file 
from  the  CD  ROM  disk  onto  the  computer's  hard  drive.  The  nature  of  the  hardware  and  the  size 
of  video  files  causes  significant  presentation  delays  -  in  some  cases  more  than  two  minutes 
while  video  files  are  being  loaded. 

Recommendation.  As  a  substitute  for  or  complement  to  video,  simple  animation  can  be 
very  cost  effective,  and  is  highly  recommended. 

Embedding  audio  in  the  lessons.  The  authoring  program  used  allowed  the  creation  of 
embedded  audio  that  was  activated  automatically  when  a  screen  image  changed.  This  use  is 
adequate  for  introductory  statements  or  information  about  the  a  lesson.  It  has  one  possible  and 
potentially  serious  disadvantage:  while  text  appearing  on  screen  can  be  read  and  re-read  by  the 
student,  embedded  audio  that  is  triggered  by  a  screen  change  "goes  by"  only  once  and  must  be 
attended  to  and  "caught"  as  it  passes.  Sometimes  it  would  of  advantage  to  the  trainee  to  be  able 
to  listen  to  an  audio  message  a  second  or  even  a  third  time. 

A  different  way  of  presenting  audio  was  developed  by  ‘detaching’  the  audio  from  the 
screen  advance  function.  An  audio  button  appears  on  the  screen,  labeled  as  “audio”  with  the 
speaker  icon  representing  sound.  The  trainee  is  told,  in  the  text  on  screen,  to  press  the  audio 
button  to  hear  the  audio  segment.  This  allows  the  trainee  to  listen  to  the  audio  when  ready,  and, 
more  importantly,  to  repeat  the  audio  as  often  as  desired.  Several  trainees  commented  favorably 
on  this  particular  aspect  of  the  use  of  audio. 

Recommendation.  Allow  the  students  to  access  audio  when  they  are  ready,  and  allow 
them  to  repeat  the  audio  segment  as  often  as  they  feel  is  necessary. 

The  final  lesson  learned  involves  a  finding  from  the  beta  testing  concerning  the  optimal 
length  of  the  CBI  lessons. 

Keeping  lessons  to  appropriate  length.  The  beta  testing  of  the  training  lanes  revealed  that 
certain  lanes  were  much  longer  than  90  minutes  and  were  associated  with  lower  post  test  scores. 
One  explanation  for  this  phenomenon  is  that  because  the  lessons  are  designed  to  actively  engage 
the  trainee,  when  the  lesson  is  longer  than  one  hour  the  trainee  will  become  fatigued,  making  it 
harder  to  learn  new  material,  and  will  also  begin  to  forget  details  learned  earlier. 

Recommendati  on .  There  may  be  some  variation  among  individuals  in  tolerance  for 
longer  lessons,  but  generally  speaking,  it  seems  to  be  a  good  policy  to  keep  lessons  very  close  to 
one  hour  in  length.  This  would  also  make  the  lessons  easier  to  schedule  around  trainees'  work 
schedules,  since  one  hour  assignments  away  from  work  are  generally  easier  to  schedule  than 
assignments  that  run  to  two  hours  or  longer. 
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Project  Oversight 

Project  oversight  was  one  of  the  most  difficult  aspects  of  this  project.  There  were  several 
stakeholders,  and  many  ways  of  passing  information  about  project  expectations.  It  was  difficult, 
at  times,  to  reconcile  the  apparently  different  points  of  view  about  project  goals.  This  section 
should  be  read  as  a  set  of  recommendations  for  improving  project  performance,  not  as  a  set  of 
complaints  or  rationalizations,  or  as  attempts  to  shift  the  responsibility  for  project  problems. 

ARPA  and  the  USARNG  sponsored  the  project.  ARI  acted  as  the  intermediary  between 
ARPA  (seen  as  the  ‘ultimate’  customer  in  this  relationship)  and  the  contractor  team.  Since  the 
contractor  was  working  for  ARI,  ARI  was  the  ‘immediate’  customer^.  ARPA  and  the  lANG 
provided  the  project  SMEs.  These  personnel  had  a  direct  line  of  communication  with  the 
ultimate  customer.  They  often  gathered  information  from  that  customer  about  project  goals  that 
appeared  to  be  inconsistent  with  that  given  by  ARPA  to  ARI,  or  passed  from  ARI  to  the 
contractor.  Two  brief  examples  will  illustrate  these  problems. 

1)  ARPA  sent  a  separate  contractor  to  Camp  Dodge  to  gather  information  from  the 
project  about  one  of  the  planned  ‘collective’  lanes.  This  contractor  was  to  develop  an  alternative 
training  module  for  this  lane  making  heavy  use  of  audio.  This  was  consistent  with  ARPA’s 
mission  to  try  highly  innovative  technologies.  The  project  SMEs  interpreted  this  to  mean  that 
they  had  been  relieved  of  the  responsibility  of  preparing  this  lane.  However,  when  the  contractor 
asked  ARI  to  ask  ARPA  if  this  was  the  case,  ARPA  said  that  they  still  expected  this  lane  to  be 
developed  as  originally  proposed. 

2)  When  there  were  unforseen  delays  in  receiving  materials  from  the  BSTS  project,  the 
SMEs  asked  the  ARPA  sponsor  for  permission  to  concentrate  efforts  on  the  lanes  training.  This 
request  was  made  on  two  occasions,  and  seemed  to  be  agreed  to.  Consequently,  there  were 
delays  in  starting  on  the  FSB  staff  training  portions  that  could  have  been  done  in  the  absence  of 
the  BSTS  materials.  Later  the  contractor  was  criticized  for  being  so  far  behind  in  the 
development  of  the  staff  training. 

Part  of  the  problem  was  due  to  the  multiple  channels  of  commimication,  rather  than 
having  one  ‘official’  chain  of  command.  Another  part  of  the  problem  was  that  most  of  the 
direction  was  oral,  very  little  was  committed  to  paper.  This  made  it  more  likely  to  be  subject  to 
multiple  interpretations  as  it  passed  from  person  to  person. 

Another  aspect  of  oversight  is  related  to  providing  timely  feedback  to  the  project  on  the 
materials  it  is  developing.  Obviously,  with  the  ARPA  and  lANG  SMEs  involved  intimately  in 
the  project,  they  could  vouch  for  the  degree  to  which  the  project  was  on  course  and  the  materials 


^  This  relationship  was  even  more  difficult  because  most  of  the  work  was  performed  by  a 
subcontractor,  whose  ‘immediate’  client  was  the  prime  contractor. 
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being  developed  were  acceptable.  On  the  other  hand,  this  might  also  have  been  a  conflict  of 
interest.  When  personnel  from  ARPA  headquarters  or  ARI  visited  the  project,  they  did  not 
comment  on  the  intermediate  products.  They  were  mostly  concerned  with  whether  the  project 
was  adhering  to  the  original  time  lines,  not  the  quality  of  the  work  being  done. 

Many  visitors,  from  a  number  of  military  organizations,  during  visits  to  Camp  Dodge  for 
other  purposes,  made  informal  visits  to  the  project  staff  buildings  to  look  at  the  lane  lessons. 

This  included  officers  representing  2d  Armor  Division,  III  Corps,  and  Second  Army,  among 
others.  Immediate,  verbal  feedback  provided  by  those  visitors  —  favorable  without  exception  -- 
was  as  informal  as  the  visits,  and  the  Camp  Dodge  contractor  staff  never  received  a  copy  of  any 
written  report  evaluating  the  intermediate  or  final  products.  The  project  staff  was,  in  effect, 
operating  in  a  vacuum,  without  the  help  that  feedback  from  external  sources  and  organizations 
can  provide. 

Eventually,  resolving  the  issues  and  misunderstandings  required  several  telephonic  and 
video  conference  calls,  and  the  preparation  of  written  documents  describing  responsibilities  more 
clearly.  Some  of  this  might  have  been  avoided  by  distributing  clearer  information  about  project 
goals  in  written  form  through  all  parallel  chains  of  command. 

Recommendations.  Having  both  contractor  and  customer  personnel  working  together 
requires  special  care  in  managing  contract  oversight.  Directions  given  to  one  group  must  be 
made  clear  (written  form  preferred)  and  passed  through  the  proper  chain  of  command  to  the 
other.  IPRs  should  focus  on  the  status  of  the  development  as  laid  out  in  the  plan  for  development 
and  the  quality  of  the  materials  being  developed.  The  necessary  trade-off  between  quality  and 
timeliness,  within  a  given  budget,  must  be  addressed  directly. 
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Original  Specification  for  FSB  Company  Training  Lanes 
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APPENDIX  B 


Introduction  to  Multimedia  Computer  Based  Instruction 


This  appendix  describes  three  aspects  of  multimedia  computer  based  instruction  (CBI): 

1)  interactive  instruction, 

2)  the  use  of  multimedia  components,  and 

3)  the  lesson/trainee  interface  —  navigating  through  the  course. 

Interactive  Instruction 

Interaction  is  the  fundamental  principle  imderlying  CBI:  The  more  the  students  interact 
with  the  material,  the  more  they  will  learn.  Students  interact  with  the  material  by  making  active 
responses  to  questions  and  problems,  and  by  receiving  feedback  on  those  responses.  The  active 
responses  require  the  student  to  process  the  information,  interpret  it,  and  apply  it.  Feedback  tells 
the  students  whether  the  response  is  correct  or  incorrect,  supplying  additional  information  about 
the  error,  or  indicating  the  correct  response. 

The  instructional  developer’s  task  is  to  design  the  presentation  of  the  material  and  the 
student  interactions  and  feedback,  called  guided  practice.  In  some  applications,  the  feedback 
may  lead  to  an  alternate  instructional  branch,  if  it  has  been  demonstrated  that  a  particular  error  is 
symptomatic  of  a  problem  requiring  another  instructional  approach’. 

There  are  a  variety  of  formats  for  eliciting  active  responses: 

Select  an  answer  from  a  list  (multiple  choice  or  true  false) 

Match  terms  or  visuals  from  two  lists 

Arrange  steps  in  the  proper  order 

Enter  short  answers  via  the  keyboard 

Select  items  from  a  set,  identifying  correct  procedures 

Evaluate  a  description  as  correct  or  incorrect 

In  multimedia  CBI  the  content  of  these  formats  may  be  text,  audio,  still  drawing  or  photo, 
animation  or  video.  For  example,  students  may  be  asked  to  place  text  or  pictorial  representations 
of  a  sequence  of  steps  in  the  proper  order.  Or,  students  may  evaluate  text  or  video 


’  The  current  project  did  not  attempt  to  do  the  extensive  testing  that  such  elaborate 
feedback  would  require.  The  evaluation  results  indicate  that  most  students  understood  the 
material  quite  well. 
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representations  of  a  procedure  as  being  correct  or  incorrect.  Matching  problems  can  be 
constructed  of  parallel  text  lists,  or  visuals,  or  combinations. 

Feedback  can  be  the  simple  statement  that  the  answer  was,  or  was  not,  correct.  Or,  it  may 
state  why.  Again,  the  multiple  media  may  be  used  to  show,  as  well  as  state,  the  correct  response. 

Multimedia  Components 

The  multimedia  capabilities  of  contemporary  desktop  computers  allow  instruction  to  be 
enhanced  with  elements  that  increase  realism  (show  rather  than  tell)  and  provide  a  richer  learning 
environment.  Using  combinations  of  media  may  facilitate  learning  by  engaging  more  of  the 
student’s  brain  in  processing  the  information,  and  may  be  especially  helpful  to  students  with 
learning  modality  preferences  other  than  printed  text. 

Briefly,  some  of  the  multimedia  components  and  their  uses  are: 

Text.  Text  is  still  one  of  the  most  efficient  ways  to  present  material  of  a  didactic  nature: 
rules,  procedural  steps,  numerical  tables,  etc.  Multimedia  CBI  can  enhance  text  (hypertext)  to 
allow  students  to  navigate  in  a  more  ad  hoc,  less  linear,  fashion  when  they  wish.  Multimedia 
CBI  can  also  assist  students  to  learn  how  to  deal  with  standard  text  references  they  must  use  (in 
the  current  project,  the  field  manual,  or  FM,  is  a  good  example). 

Audio.  The  combination  of  voice  with  music  or  sound  effects  is  in  general  use  as  a  form 
of  introduction.  Such  uses  are  primarily  cosmetic  or  attention-getting  applications.  In 
multimedia  CBI,  audio  can  be  used  to  present  information,  direct  attention  to  something  on  or  off 
the  screen,  and  guide  actions  at  the  computer,  or  nearby. 

As  a  means  of  presenting  information  it  is  instructive  to  note  that  the  spoken  word  is 
typically  the  basis  of  most  television  news  programs.  The  graphics  supplement  the  verbal 
description,  but  rarely  displace  it. 

One  use  of  audio  that  should  be  avoided  is  to  use  a  voice  to  repeat,  word  for  word,  the 
text  information  that  appears  on  the  screen.  Judicious  use  of  audio  makes  it  effective  in 
providing  emphasis  on  critical  content  and  key  points,  and  increasing  the  trainee's  concentration 
on  the  information  provided.  This  effect  probably  diminishes  if  audio  is  used  too  fi-equently,  or 
for  less  critical  content. 

One  aspect  of  audio  one  that  is  not  characteristic  of  visual  information  that  should  be 
capitalized  upon  wherever  possible  is  the  non-directional  nature  of  sound  reception.  In  order  to 
receive  visual  or  text  information  from  the  computer,  the  trainee  has  to  be  looking  at  the  screen, 
but  audio  information  is  received  by  the  human  ear  no  matter  where  the  trainee  is  looking  or 
what  he  or  she  is  doing.  The  audio  channel,  for  example,  could  guide  a  trainee  through  the  tasks 
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of  filling  out  a  form,  reading  a  map,  finding  something  in  a  reference,  or  performing  a  variety  of 
tasks,  at  or  away  from  the  computer. 


Drawings  and  Photos.  Drawings  and  photos  may  provide  all  of  the  graphic  information 
that  is  needed  to  reinforce  the  learning  from  the  text  and  audio  modalities.  Judicious  choices 
must  be  made  between  still  drawings,  still  photographs,  animations  and  video.  In  general,  any 
form  of  motion  costs  more  in  time  and  labor  to  produce,  and  requires  more  storage  capacity  than 
stills. 


Videos  may  be  a  useful  source  of  still  images,  however.  It  is  often  relatively  economical 
to  extract  a  still  from  a  video,  rather  than  prepare  a  separate  graphic. 

Photos  are  generally  more  realistic  than  drawings.  But  photographs  are  susceptible  to 
problems  Avith  contrast,  shadows  and  reflectance  that  may  make  a  photo  image  ineffective  for 
training.  Professional  photographers  know  how  to  avoid  these  problems,  but  may  be  too 
expensive  for  a  particular  project.  Drawings  can  be  competitive  in  cost  compared  to  professional 
photos. 


There  are  resources  for  photos  and  drawings.  In  the  current  setting,  there  were  many  line 
drawings  that  could  be  extracted  from  Army  manuals.  Even  clip  art  collections  could  be  used  by 
coloring  the  black  and  white  images,  adding  backgrounds,  and  in  some  cases,  elements  such  as 
uniforms  and  weapons  that  ‘militarized’  the  image.  No  more  than  20  percent  of  the  still  images 
for  this  project  were  developed  from  scratch. 

Animation  and  Video.  One  of  the  primary  values  of  video  is  to  show  realistic  movements 
and  actions  that  the  trainees  are  to  learn,  where  timing  and  motions  are  critical.  Several  of  the 
medical  lessons  made  effective  use  of  video  for  this  purpose,  showing  how  certain  medical  and 
treatment  procedures  are  carried  out.  To  a  lesser  degree,  the  maintenance  lessons  benefited  from 
video  for  similar  purposes. 

Another  use  of  animation  or  video,  like  audio,  is  for  more  cosmetic  (as  opposed  to 
instructional)  purposes:  to  introduce  a  topic  or  attract  attention. 

Animation  is  typically  easier  and  more  economical  to  provide  than  video.  It  focuses 
attention  on  the  most  essential  elements.  As  the  degree  of  realism  required  to  make  the 
instruction  effective  grows,  the  costs  of  animation  (e.g.  fully  articulated  human  motion  on  a 
realistic  background)  may  prove  to  be  as  expensive  as  shooting  video.  For  example,  to  introduce 
a  lesson  concerning  properly  grounding  a  fuel  tank-trailer,  an  animation  was  created  to  show  the 
tank-trailer  delivered  by  a  tractor.  Video  taping  the  same  scene  would  have  been  very  costly. 

The  Lesson/Trainee  Interface:  Navigating  through  the  Course 
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Although  each  lesson  varies  somewhat  in  its  design  and  appearance,  the  basic 
components  and  functions  are  the  same.  When  a  trainee  enters  a  lesson,  the  first  screen  offers  the 
initial  set  of  options,  consisting  of  information  about  the  lesson  or  about  each  of  the  topics  in  the 
lesson.  Each  of  the  options  provides  only  information:  this  menu  does  not  lead  directly  into  the 
interactive  lessons  themselves.  The  trainee  uses  a  mouse  (or  similar  pointing  device)  to  ‘click’ 
on  the  selection  he  or  she  desires. 

At  the  bottom  of  every  screen  (except  for  special  purpose  screens,  such  as  questions, 
matching  items,  etc.)  five  clickable  ‘buttons’  are  provided  for  navigational  purposes: 


Previous 

Menu  j 

Options 

Quit 


Next 


Figure  1 .  Navigation  button  template. 

The  most  frequently  used  button  is  the  Next  button,  which  displays  the  next  screen  in  the 
sequence.  The  Previous  button  moves  back  in  the  sequence,  sometimes  to  the  immediate 
previous  screen,  but  sometimes,  due  to  the  IconAuthor  structuring,  to  the  screen  at  the  beginning 
of  the  topic  sequence.  The  Quit  button  does  exactly  that,  quitting  the  sequence  and  exiting  the 
lesson.  The  Menu  button  presents  the  menu  screen  shown  in  Figure  2.  The  Options  button 
presents  a  menu  of  options,  the  principle  one  of  which  is  a  glossary  of  technical  and  military 
terms,  abbreviations,  and  acronyms. 

After  selecting  information  options  from  the  opening  screen,  the  trainee  clicks  on  either 
the  Next  button  or  the  Menu  button,  which  presents  the  screen  shown  in  Figure  2. 
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The  Take  Post  Test  button  in  the  upper  right  comer  of  the  screen  allows  trainees  to  move 
directly  to  the  post  test  if  they  feel  they  have  already  mastered  all  of  the  required  competencies. 

Before  the  trainee  can  select  a  topic,  a  small  window  appears  asking  the  trainee  whether 
he  or  she  wants  to  take  the  lesson  "for  record,"  or  simply  to  review  the  material  in  a  "browse" 
mode.  The  trainee  clicks  on  either  choice,  then  clicks  on  the  desired  topic.  Each  of  the  topic 
buttons  moves  directly  into  the  instmctional  sequence  for  that  topic.  The  trainee  moves  through 
a  topic,  answering  questions  or  carrying  out  tasks,  and  eventually  taking  a  quiz  at  the  end  of  the 
topic. 


The  "browse"  mode  was  included  in  the  lessons  at  the  request  of  AREA.  There  is  no 
difference  between  the  browse  and  “for  record”  modes  in  terms  of  what  the  trainee  sees  and  does 
during  the  lesson.  The  difference  is  that  the  management  system  does  not  keep  records  on  the 
performance  of  browsers.  Trainees  or  others  who  want  to  look  over  the  lessons  to  see  what  they 
contain  can  easily  do  so;  soldiers  who  want  to  obtain  specific  items  of  information  that  might  be 
more  difficult  to  find  in  their  manuals  can  use  the  browse  mode  for  that  purpose.  The  browse 
mode  can  also  be  used  for  review  and  refresher  use,  although  for  those  sometimes  essential 
purposes,  the  lessons  might  more  appropriately  be  taken  for  the  record. 

Smnmary 

The  combination  of  interactive  instruction  with  multimedia  components  in  an 
instructional  environment  that  permits  unlimited  browsing  and  review  should  engage  the  students 
effectively  and  allow  them  to  progress  to  skill  mastery.  The  formative  evaluation  of  the  lessons 
prepared  for  this  project  illustrates  the  effectiveness  of  this  technology. 
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APPENDIX  C 

Original  Specification  for  FSB  Staff  Lessons 
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APPENDIX  D 

STAFF  TRAINING  VALIDATION 

INTRODUCTION 


Background 

The  Advanced  Research  Projects  Agency  (ARP A)  and  the  Army  National  Guard 
(ARNG)  contracted  with  BDM  Federal,  Inc.  to  develop  a  series  of  computer-based  courses  to 
instruct  Forward  Support  Battalion  (FSB)  and  Support  Battalion  (SB)  staff  officers.  These 
courses  were  to  be  based  upon  similar  courses  developed  for  the  battle  staffs  of  Armor  and 
Mechanized  Infantry  battalions  in  a  parallel  effort  sponsored  by  ARPA  and  the  ARNG. 

The  staff  of  the  FSB  is  organized  somewhat  differently  from  that  of  a  combat  battalion. 
One  person  serves  in  the  combined  roles  of  S2  and  S3  and  there  is  an  additional  officer,  the 
Support  Operations  Officer  (SPO).  A  Support  Battalion  for  a  separate  brigade  has  all  of  the  staff 
positions  of  a  combat  battalion,  plus  a  Battalion  Materials  Management  Officer  (BMMO).  The 
courses  fi-om  the  Battle  Staff  Training  System  (BSTS)  had  to  be  modified  to  adapt  them  to  the 
duties  of  the  corresponding  staff  position  in  the  FSB  or  SB.  The  courses  for  the  SPO  and  the 
BMMO  were  developed  by  the  ARPA,  ARNG  and  contractor  staff  at  Camp  Dodge,  Iowa. 

The  BSTS  project  commenced  at  the  same  time  as  the  project  to  develop  training  for  the 
FSB  and  SB.  The  courses  received  from  the  BSTS  became  available  late  in  the  development 
cycle  for  the  FSB  project.  To  meet  project  deadlines,  BSTS  courses  that  had  not  been  validated 
were  used  as  the  basis  for  making  modifications  to  suit  the  FSB  and  SB  staffs.  The  validation 
information  presented  here  concerns  only  the  versions  of  the  courses  as  adapted  to  the  FSB  and 
SB  staffs. 

There  are  two  types  of  lessons  in  the  staff  courses;  1)  Fully  CBI  lessons,  in  which  the 
instruction  is  done  via  interaction  with  the  computer,  and  2)  text  lessons,  in  which  the  instruction 
is  done  by  having  the  student  read  Army  doctrinal  materials.  There  is  a  computer  managed 
component  of  both  lessons,  consisting  of  a  pretest,  posttest  and  a  practical  exercise  for  each 
lesson.  This  component  allows  student  progress  to  be  entered  into  the  training  management 
system,  and  gives  the  student  an  opportunity  to  exercise  the  skills  he  has  attained  through  the 
instruction. 

The  formative  evaluation  of  the  staff  lessons  was  somewhat  different  from  that  described 
for  the  FSB  Lanes  Training  development  (Keesling,  1995).  The  design  of  the  courses  was 
essentially  ‘given’  in  the  materials  developed  by  the  BSTS  project.  However,  as  ARPA  staff  at 
Camp  Dodge  adapted  the  lessons  to  the  FSB/SB  some  were  greatly  changed,  while  others  were 
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only  slightly  modified.  In  some  cases  the  lessons  were  recombined  to  suit  the  FSB/SB 
instructional  needs. 

For  the  SPO  and  BMMO  courses  the  formative  evaluation  began  in  the  design  phase. 
Subject-matter  experts  and  instructional  designers  examined  the  content  and  sequencing  to 
determine  if  they  were  appropriate  on  a-priori  grounds.  The  decision  was  made  to  develop  a 
lesson  as  fully  CBI  or  text.  The  CBI  lessons  went  through  a  phase  of  ‘Alpha  testing’  in  which 
the  developers  performed  a  trial-run  of  the  lesson  to  determine  if  the  courseware  tracked  with  the 
storyboards,  and  to  identify  flaws  in  the  logic  of  presentation  and  errors  or  ambiguities  in  the 
instructions  for  navigation  through  the  course. 

All  of  the  lessons  for  all  staff  positions  went  through  a  phase  of  ‘Beta  testing’  in  which 
subjects  took  the  lessons  in  their  near-final  form  to  determine  if  they  could  navigate  through  the 
course  successfully  and  the  extent  to  which  they  learned  from  the  material.  Errors  detected  in  the 
content  or  in  the  navigation  directions  were  corrected  in  the  final  version  of  the  lesson. 

Purpose 

This  report  documents  the  validation  testing  performed  on  the  staff  lessons  developed  for 
the  FSB  and  SB.  The  report  documents  the  performance  gains  attained  by  the  test  subjects  and 
the  typical  amounts  of  time  required  to  complete  the  lessons. 


ANALYSIS  OF  BETA  TEST  DATA 


Data 


Exhibit  D-1  lists  the  data  available  for  this  validation  study.  There  were  12  fully  CBI 
lessons  and  27  text  lessons.  It  was  difficult  to  find  a  large  pool  of  subjects  to  test  these  lessons. 
The  ARPA  staff  conducted  the  testing  among  themselves  to  facilitate  project  completion. 
Thirty-one  of  the  lessons  had  only  one  beta  test,  two  lessons  had  two  beta  tests  and  one  had  four 
beta  tests.  In  total,  48  beta  test  records  were  obtained*. 

Analysis 

The  results  will  be  presented  in  two  sections,  one  for  the  fully  CBI  lessons  and  one  for 
the  text  lessons.  Each  section  will  discuss  the  percentage  of  students  who  attained  mastery 


*  Since  the  first  lesson  for  the  SPO  was  identical  to  the  first  lesson  for  the  BMMO,  the 
two  records  for  the  SPO  were  averaged  and  a  synthetic  record  with  these  averages  was  inserted 
for  BMMO  lesson  one. 
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(defined  as  80  percent  or  better  correct  on  the  posttest),  the  average  pretest  and  posttest  scores, 
and  the  amount  of  time  required  to  take  the  lesson. 


CBI  Lessons.  There  were  1 5  beta  tests  performed  on  the  1 2  CBI  lessons®.  Four  of  these 
students  obtained  pretest  scores  of  80  percent  or  higher.  Of  the  other  1 1  records,  10  (or  91 
percent)  attained  mastery  level  on  the  posttest.  Table  1  shows  the  average  pretest  and  posttest 
attainment  of  these  students.  These  scores  confirm  that  the  subjects  who  were  non-masters  at  the 
time  of  the  pretest  learned  a  considerable  amount  of  information  from  these  lessons.  Even  the 
subjects  who  were  masters  at  the  pretest  improved  their  scores  substantially. 

Table  1. 

Pretest  and  Posttest  average  scores  for  fully  CBI  lessons. 


Pretest  Non-Masters 
N=ll 

Pretest  Masters 

N  =  4 

Average  Pretest  Score 

64.1  (9.2) 

83.7(3.3) 

Average  Posttest  Score 

84.4  (6.3) 

93.5  (8.5) 

Numbers  in  parentheses  are  standard  deviations. 


On  average,  it  took  58  minutes  to  complete  each  of  these  fully  CBI  lessons,  including  the 
pretest  and  posttests.  The  lessons  themselves  took  about  32  minutes,  on  average.  These  times 
should  allow  National  Guard  staff  officers  to  take  these  lessons  during  weekend  drill  periods. 

Text  Lessons.  There  were  32  beta  test  records  on  the  27  text  lessons'®.  These  records 
were  obtained  from  subjects  who  took  the  pretest,  performed  the  practical  exercise,  and  took  the 
posttest.  They  did  not  follow  the  instructions  to  read  the  Army  doctrinal  materials.  The  purpose 
of  these  beta  tests  was  to  examine  the  functionality  of  the  courseware  managing  the  instruction 
and  testing. 

Although  it  may  seem  irrelevant  to  examine  the  scores  obtained  by  these  subjects,  they 
are  relevant  to  the  use  of  these  lessons  as  refresher  materials.  The  subjects  who  performed  the 
beta  testing  were  generally  knowledgeable  in  the  area  of  staff  functions.  One  was  a  Lieutenant 
Colonel,  one  a  Major  and  one  a  Captain,  all  of  whom  had  served  on  battalion  staffs  in  the 


®  One  additional  beta  test  record  did  not  have  a  pretest  score  and  was  not  counted  in  the 
following  analyses.  The  student  attained  mastery  on  the  posttest  (87  percent  correct). 

'®  There  was  one  additional  record  that  did  not  have  a  pretest  recorded  and  was  not 
included  in  these  analyses.  The  subject  performing  this  beta  test  obtained  a  score  of  80  percent 
correct  on  the  posttest. 
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ARNG.  These  subjects  generated  12  records  indicating  that  they  were  masters  at  the  time  of  the 
pretest.  Of  the  20  other  records,  12  (or  60  percent)  indicated  attainment  of  mastery  by  the  time 
of  the  posttest.  Recall  that  the  text  lessons  contained  a  practical  exercise  as  well  as  the  pretest 
that  could  contribute  to  the  student’s  orientation  to  and  understanding  of  the  material.  Further 
examination  of  the  records  showed  that  three  of  the  records  indicated  that  the  beta  test  subject  did 
not  take  the  practical  exercise.  In  all  three  of  these  instances  the  subject  did  not  attain  mastery  on 
the  posttest.  When  these  records  are  set  aside,  12  out  of  17  (or  70  percent)  of  the  pretest 
nonmasters  attained  mastery  by  the  time  of  the  posttest.  This  evidence  supports  the  use  of  these 
lessons  as  refreshers  for  knowledgeable  staff  members. 

Table  2  shows  the  average  pretest  and  posttest  scores  for  the  beta  test  records  for  the  text 
lessons.  These  averages  confirm  that  the  lessons  could  be  used  as  refresher  materials.  In  order  to 
validate  the  entire  lesson,  subjects  that  have  little  staff  experience  would  have  to  be  instructed  to 
take  the  entire  lesson,  including  reading  the  textual  materials. 

Table  2. 

Pretest  and  Posttest  average  scores  for  text  lessons. 


Pretest  Non-Masters 

Did  Not 

Take 

Took 

Practical 

Practical 

Exercise 

Exercise 

Pretest  Masters 

N  =  3 

N=17 

N=12 

Average  Pretest  Score 

61.7  (6.8) 

66.0  (8.7) 

84.3  (6.2) 

Average  Posttest  Score 

73.7(1.2) 

80.2  (8.1) 

95.3  (4.9) 

Numbers  in  parentheses  are  standard  deviations. 


It  took  the  subjects  who  were  not  masters  at  pretest  about  36  minutes  to  complete  the 
pretest,  practical  exercise,  and  posttest.  This  brief  period  of  time  should  be  relatively  easy  to 
incorporate  into  a  training  weekend  for  ARNG  officers.  It  is  not  possible  to  predict  the  amount 
of  time  required  to  read  the  textual  materials.  The  CBI  system  allows  the  officer  to  read  these  on 
his  own  and  return  to  finish  the  lesson  at  a  later  time.  This  should  facilitate  use  of  the  complete 
text  lessons  by  new  staff  officers. 


CONCLUSION 
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The  CBI  lessons  developed  for  the  FSB  and  SB  staff  officers  offer  an  opportunity  to  learn 
important  concepts  in  a  reasonably  brief  span  of  time.  The  text  lessons  function  appropriately 
and  may  be  used  without  the  text  material  as  a  refresher  for  officers  with  some  experience.  It  is 
assumed  that  since  the  text  material  is  from  Army  doctrinal  sources  it  will  be  appropriate  to  train 
the  less  sophisticated  officers  to  a  state  of  subject  mastery. 
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EXHIBIT  D-1 


Beta  Test  Records  for  FSB/SB  Staff  Lessons 


POSITION 

LESSON 

PRETEST 

POSTTEST 

PRETIME 

LSNTIME 

POSTTIME 

BMMO 

BMl 

70.0 

90.0 

13.0 

3G.0 

13.0 

BMMO 

BM2 

8G.0 

8G.0 

10.0 

15.0 

10.0 

BMMO 

BM2 

71.0 

8G.0 

10.0 

10.0 

10.0 

BMMO 
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